WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
G11B 31/00, G06F 15/44 



Al 



(11) International Publication Number: 
(43) International Publication Date: 



WO 97/15926 

1 May 1997(01.05.97) 



(21) International Application Number: PCT/EB96701056 

(22) International Filing Date: 7 October 1996 (07.10.96) 



(30) Priority Data: 
115552 
60/008,874 



8 October 1995 (08.10.95) IL 
19 December 1995 (19.12.95) US 



(71) Applicant (for all designated States except US): FACE IMAG- 

ING LTD. [ILTL]; 23 Hillel Street, Jerusalem 94581 (IL). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): PELEG, Shmuel [ILflLJ; 
45 Bar-Cochva Street, Jerusalem 97892 (IL). COHEN, Ran 
PUIL]; 6 Yekotiel Adam Street, Petach Tikva 49326 (IL). 
AVNIR, David (IUIL]; 2 Novomeiski Street, Jerusalem 
96908 (IL). 

(74) Agent: NOAM, Meir, c/o Hauptman, Benjamin, J., Lowe, 
Price, Leblanc & Becker, Suite 300, 99 Canal Center Plaza, 
Alexandria, VA 22314 (US). 



(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR, 
BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, FI, GB, GE. 
HU, IL, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, LS, 
LT. LU. LV, MD, MG, MK, MN, MW, MX, NO, NZ, PL, 
PT, RO, RU, SD, SE, SG, SI, SK, TJ, TM, TR, TT, UA, 
UG. US, UZ, VN, ARIPO patent (KE, LS, MW, SD, SZ, 
UG), Eurasian patent (AM, AZ, BY, KG. KZ. MD. RU, TJ, 
TM). European patent (AT. BE, CH, DE. DK, ES, FI. FR. 
GB, GR, IE, IT, LU, MC, NL, PT, SE), OAPI patent (BF, 
BJ, CF, CG, CI, CM, GA, GN, ML, MR, NE, SN, TD, TG). 



Published 

With international search report. 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 



(54) Tltie: A METHOD FOR THE AUTOMATIC COMPUTERIZED AUDIO VISUAL DUBBING OF MOVIES 



(57) Abstract 

A method using computer software for automatic au- 
dio visual dubbing (5), using an efficient computerized auto- 
matic method for audio visual dubbing of movies by comput- 
erized image copying of the characteristic features of the lip 
movements of the dubber onto the mouth area of the original 
speaker. The invention uses a method of vicinity-searching, 
three-dimensional head modeling of the original speaker (3), 
and texture mapping (10) technique to produce new images 
which correspond to the dubbed sound track. The invention 
thus overcomes the well known disadvantage of the correla- 
tion problems between lip movement in an original movie and 
the sound track of the dubbed movie. 
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A METHOD FOR THE AUTOMATIC COMPUTERIZED AUDIO VISUAL 

DUBBING OF MOVIES 

FIELD OF THE INVENTION 

The present invention relates to a method for automatic 
audio visual dubbing. More specifically the 

said invention relates to an efficient computerized 
automatic method for audio visual dubbing of movies by 
computerized image copying of the characteristic features 
of the lips movements of the dubber onto the mouth area 
of the original speaker. The present invention uses a 
method of vicinity-searching, three-dimensional head 
modeling of the original speaker, and texture mapping 
techniques in order to produce the new images which 
correspond to the dubbed sound track. 

The invention overcomes the well known disadvantage of 
the correlation problems between lips movement in the 
original movie and the sound track of the dubbed movie. 

DEFINITIONS OF TERMS RELATED TO THE INVENTION 

First are provided some definitions of" important key 
words employed in this specification. 

Actor (the original actor) - an actor, speaker, singer, 
animated character, animal, an object in a movie, or a 
subject in a still photograph. 
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Audio Visual Dubbing - Manipulating, in one or more 
frames, the mouth area of the actor so that its status 
will be similar as much as possible to that of the dubber 
in the reference frame. 

Correlation Function - a function describing the 
similarity of two image regions. The higher the 
correlation, the better is the match. 

Dubber - The person or persons, who 
speak/narrate/sing/interpret the target text. The dubber 
can be the same as the actor. 

Dubbing - Replacing part or all of one or more of the 
original sound tracks of a movie, with its original text 
or sounds (including the case of the silent track of a 
still photograph), by another sound track containing the 
target text and/or sound. 

Edge Detector - A known image processing technique used 
to extract boundaries between image regions which differ 
in intensity and/or color. 

Pace Parametrization - a method that numerically 
describes the structure, location, and expression of the 
face. 
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Head Model - A three-dimensional wire frame model of the 
face that is controlled by numerous parameters that 
describe the exact expression produced by the model (i.e. 
smile, mouth width, jaw opening, etc.)- 

Movie (the original movie) - Any motion picture (e.g. 
cinematic feature film, advertisement, video, animated 
cartoon, still video picture, etc.). A sequence of 
consecutive pictures (also called frames) photographed in 
succession by a camera or created by an animator. In the 
case of the movie being a still photograph, all of the 
consecutive pictures are identical to each other. When 
shown in rapid succession an illusion of natural motion 
is obtained, except for the case of still pictures. A 
sound-track is associated with most movies, which 
contains speech, music, and/or sounds, and which, is 
synchronized with the pictures, and in particular where 
the speech is synchronized with the lip movements of the 
actors in the pictures. Movies are realized in several 
techniques. Common methods are: (a) recording on film, 
(b) recording in analog electronic form ("video"), (c) 
recording in digital electronic form, (d) recording on 
chips, magnetic tape, magnetic disks, or optical disks, 
and (e) read/write by magnetic and/or optical laser 
devices. Finally, in our context, an "original movie" is 
also an audio-visual movie altered by the present 
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invention, which serves as a base for further 
alterations . 

Original Text - A text spoken or sung by the actor when 
the movie is being made, and which is recorded on its 
sound track. The text may be narrated in the background 
without showing the speaker, or by showing a still 
photograph of the speaker. 

Pixel - Picture element. A digital picture is composed of 
an array of points, called pixels. Each pixel encodes the 
numerical values of the intensity and of the color at the 
corresponding picture point. 

Reference Similarity Frame - a picture (being a frame 
in the original movie, a frame in any other movie, or a 
still photograph) in which the original actor has the 
desired features of the mouth-shape and head posture 
suitable for the audio visually dubbed movie. 

Target Text - A new vocal text, to replace the original 
vocal text of the actor. The target text may also be that 
which is to be assigned to an actor who was silent in the 
original movie. The new text can be in another language, 
to which one refers as DUBBING. However, this invention 
relates also to replacement of text without changing the 
language, with the original actor or with a dubber in 
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that same language. The target text may have the same 
meaning as the original text, but may have also a 
modified, opposite, or completely different meaning. 
According to one of many applications of the present 
invention, the latter is employed for creation of new 
movies with the same actor, without his/her/its active 
participation. Also included is new vocal text used to 
replace the null vocal text attached to one or more still 
photographs . 

Texture Mapping - a well known technique in computer 
graphics which maps texture onto a three-dimensional wire 
frame model. 

Two-Dimensional Projection - The result of the rendering 
of the three-dimensional face model onto a two- 
dimensional device like a monitor, a screen, or 
photographic film. 

BACKGROUND OF THE INVENTION 

Movies are often played to an audience that is not 
familiar with the original language, and thus cannot 
understand the sound track of such movies. Two well known 
common approaches exist to solve this problem, m one 
approach sub-titles in typed text of the desried language 
are added to the pictures, and the viewers are expected 



WO 97/15926 



6 



PCT/IB96/01056 



to hear the text in a foreign language and simultaneously 
to read its translation on the picture itself. Such 
reading distracts the viewers from the pictures and from 
the movie in general. Another approach is dubbing, where 
the original sound-track with the original text is being 
replaced by another sound-track with the desired 
language, in this case there is a disturbing mis-match 
between the sound-track and the movements of the mouth. 

There have been some earlier attempts to overcome these 
disadvantages, none of which have been commercialized 
because of inherent principal difficulties which made the 
practical execution unrealistic. Thus, in U.S. Patent 
Number 4,600,281 a method is described which performs the 
measurements of the shape of the mouth manually by a 
ruler or with a cursor, and corrects the mouth- shape by 
moving pixels within each frame. As will be seen in the 
description of the invention, the method according to the 
present invention is inherently different and much 
superior in the following points: m the present 
invention the tracking of the shape of the mouth is done 
automatically and not manually, m the present invention 
changing the shape of the mouth is done by using a three- 
dimensional head model, for example like those described 
by P.Ekman and W.V.Friesen, (Manual for the Facial Action 
Unit System, Consulting Psychologist Press, Palo Alto 
1977). m the present invention the mouth area of the 
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actor is replaced using the mouth area of a reference 
similarity frame. In the present invention mouth status 
parameters of the dubber are substituted for mouth status 
parameters of the actor. 



The U.S. Patent Number 4,260,229 relates to a method of 
graphically creating lip images. This U.S. patent is 
totally different from the present invention: In the U.S. 
Patent, speech sounds are analyzed and digitally encoded. 
In the present invention no sound analysis is done; nor 
is any required at all. 

To make for better viewing of the audio visually dubbed 
movie, the present invention provides a computerized 
method wherein, in addition to replacing the sound track 
to the target text, the mouth movements of the actor are 
being automatically changed to match the target text. The 
new mouth movements are linguistly accurate and visually 
natural looking according to all of the observable 
parameters of the actor's face. 

SUMMARY OF THE INVENTION 



The present invention provides a method for automated 
computerized audio visual dubbing of movies, comprising 
of the following steps (see Pig. i) : 
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(a) selecting from the movie a frame having a picture 
preferably frontal, of the actor's head and, if 
available, a frame with its side profile; 

(b) marking on the face several significant feature 
points and measuring their locations in the frame; 

(c) fitting a generic three-dimensional head model to the 
actor's two-dimensional head picture by adapting the data 

of the ^nificant feature points, . as measured in stage - 
(b), to their location in the model; 

(d) tracking of the said fitted three-dimensional head 
model parameters throughout the movie, from one frame to 
its successor, iteratively in an automated computerized 
way and creating a library of reference similarity 
frames . 



-(e) taking a movie of a dubber wherein the dubber speaks 
the target text; 



(f) repeating stages (a,, (b, , (c, ,- and (d) with the 
dubber; 



(g) normalizing the dubber 's minimum and maximum values 
of each parameter to the actor's minimum and maximum 
valuse of the same parameters; 



WO 97/15926 



9 



PCT/IB96/01056 



(h) mapping, on a frame to frame basis, the two- 
dimensional actors face onto its three-dimensional head 
model by using a texture mapping technique, making use of 
reference similarity frames; 

(i) changing the texture mapped three-dimensional model 
obtained in stage (h) by replacing, on a frame to frame 
basis, the original mouth parameters with the mouth 
parameters as computed in stage (d) for the dubber and 
obtaining the parametric description for the new picture, 
with identical values to the original, except that the 
actor's mouth status resembles the mouth status of the 
dubber; 

(j) texture mapping the lips area of the same actor from 
a frame in the movie, with identical or very similar 
mouth status to the desired new mouth status, onto the 
lips area of the actor's head model for the current frame 
and then projecting the lips area from the actor's head 
model onto the current new frame. (This stage is 
optional, depending on the application.) 

By using the three dimensional head model one can control 
the audio visual dubbing process even if the actor is 
moving his. head. In most applications about 15 
significant feature points on the face are used in the 
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tracking stage, such as eye corners, mouth corners, and 
the nostrills. only those feature points which are 
visible to the viewer (using the information available to 
the model) are tracked. 

In the present invention, audio visual dubbing is 
normally used in conjunction with the use of audio 
dubbing; but one may also use it in conjunction with an 
audio track where no equivalent track exists in the 
original movie. 

The method according to the present invention is useful 
for the audio visual dubbing of motion pictures such as 
cinematic feature films, advertisments , video, and 
animated cartoon. Also the audio visual dubbing of still 
Photographs, wherein all of the frames of the movie are 
the same, is made possible by the present invention. For 
instance, still photographs are used for this type of 
movie in T.V. news programs where the reporter's voice is 
heard while a still photograph of him/her is shown. 

Thus, according to the present invention even speachless 
actors, infants, animals, and inanimate objects can be 
audio visually dubbed to speak in any language. 

According to our invention, the animation process saves 
much of the labor associated with the animation of the 
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mouth area. 



The present invention further provides a computer program 
(see Appendix l) for operating the computerized audio 
visual dubbing. 

The present invention further relates to the library of 
reference similarity frames as created in step d (above). 

DETAILED DESCRIPTION OF THE INVENTION 

Given an original movie where an actor speaks the 
original text, a movie of a dubber is made where the 
dubber speaks the target text in either another 
language or the same language. The movie of the dubber 
is taken while the dubber performs a routine dubbing 
adaptation of the original into the target text. 

This invention provides a method for the changing of the 
actor's facial motions in the original movie to create a 
new movie having the sound-track in the target text from 
the movie of the dubber, while the pictures are of the 
original movie, whereas the motion of the mouth of the 
actor is modified to correspond to the new sound-track. 

For brevity considerations, the description of this 
invention uses pictures in electronic digital form 
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(composed of an array of pixels), but movies in any other 
form are treatable as well. In these cases, the movie is 
translated to a digital form by existing techniques, 
manipulated in the digital form, and returned back to any 
desired form by known techniques. 

A facial expression can be described by an "action unit", 
for example the Facial Action Coding System (FACS) by 
Ekman and Friesen (Ekman et. al.)- Action units (AU) 
stands for a small change in the facial expression which 
depends on a conscious activation of muscles (H. Li, p. 
Roivainen, R. Forchheimer, 3-D Motion in Model-Based 
Facial Image Coding, IEEE Transactions in PAMI, 15 (2), 
545—555 (1993)). The AU information is expressed in 
parameter form. Using the AU parameters, many facial 
expressions can be controlled. Parameters like face 
location and size, aspect-ratios of face regions, 
location of specific face fetures and many more. 

As explained above, one of the stages of this invention 
is a three-dimensional parameterisation of the face. An 
example for one such model is the model of Parke 
(Fredric I. Parke, Parameterized Models for Fatial 
Animation, IEEE computer Graphics and Applications, 12 
(11), 61-68, (1982)) which consists of about 25 
parameters. Face parameters can be roughly divided into 
three main classes: structure parameters, location 
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parameters, and expression parameters. 

Structure parameters are fixed for every head and include 
distance ratios between the mouth and the eyes, the mouth 
and the chin, width of the model, jaw width, etc. The 
location parameters are, for example: three parameters 
for three-dimensional rotation in space and three 
parameters for three-dimensional translation (position in 
the real world). The expression parameters are, for 
instance: mouth width, smile (as an example, the 
parameter values here may be o.o for a very sad mouth and 
l.o for a very happy mouth), jaw opening, upper lip 
lifting, lower lip lowering, lip thickness, and so on. 

Using a face model, the present invention is centered on 
a computer program (see Appendix 1), which automaticly 
re-shapes the lip movements of the actor according to the 
lip movements of the dubber by searching for the nearest 
reference similarity frames. This computer program 
(software), or similar, is an integral and an important 
part of the present invention. The process, according to 
the present invention, is divided generally into the 
TRACKING phase and the NEW-MOVIE generation phase, as 
follows : 



I. TRACKING phase 



Step i : The first step is personalizing the generic 
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three-dimensional face model for both the actor and the 
dubber. In order to modify the generic face model to fit 
a specific face, some additional information is needed. 
The generic model has to be translated, scaled and 
stretched to fit the given actor's face, from its initial 
position and setting. This is done by manually pointing 
using a pointing device such as a mouth, a touch screen, 
etc., several facial feature points on the actor's face, 
e.g. eye corners, mouth corners, top and bottom of face. 
Typically a total of approximately 15 feature points are 
used, but this number may vary according to 
specifications. These feature points are marked on one 
(any) of the frames in the movie, in which the actor, 
preferably, faces the camera. The computer program then 
calculates automatically the exact model parameter's 
modifications needed for its two-dimensional projection 
to fit the actor face on the movie frame. In addition to 
using the facial feature points and in order to increase 
accuracy, the model is also adjusted to match the head 
edges, which are computed using an edge detector. If a 
side view of the actor is available it can be used to set 
several depth parameters, such as face depth and nose 
length. Otherwise, the face depth is scaled by some 
predetermined scale which is set experimentally. 



Step 2: After the generic face model has been 
personalized to the desired actor, face features in 
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several key frames in the movie are marked. The number of 
such frames can vary from a single first frame to about 
5% of all frames, depending on the difficulty of the 
segment fitting the model to the actor using the marked 
facial features in those key frames achieves a 
stabilization of the automatic tracking (described 
later), and these key frames assures stable and 
continuous tracking. Next, the program calibrates 
according to several examples of mouth shapes, later to 
be used for mouth tracking. Finally, the range of the 
mouth parameters (mimimum and maximum values) for the 
specific actor are estimated using all the values of the 
model parameters fitted to all key frames. 

Step 3: The next stage is the automatic tracking of the 
actor's face throughout the entire movie: This is 
performed from one frame to its successive frame, using 
the face model, in two steps: first, the two-dimensional 
face of the actor is mapped onto the three-dimensional 
face model using a texture-mapping technique. The model 
can now be altered by changing its parameters only, 
creating new, synthetic images, which are otherwise very 
similar to the original movie frames: everything remains 
unchanged except for different face location, its 
orientation, and its expressions. By using a "minimization 
algorithm, either analytic or numerical (such as steepest 
descent algorithm), the program now computes those 
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parameters which maximize the correlation function 
between the face area of the actor in the next frame and 
the synthesized projection of the texture-mapped face 
model. The steepest descent algorithm increases or 
decreases parameters in the direction that increases the 
correlation function. It can either work for each 
parameter separately (until it maximizes the 
correlation), or it can modify all the parameters at 



once. 



Step 4: After the model is locked on the head of the 
actor in the next frame, the mouth has to be tracked. 
This is done by first, checking the parameters of all of 
the mouths in the key frames and in several previous 
frames already tracked. Then, the frame that gives the 
higher correlation is choosen as a first guess for the 
tracking. Next, the same minimization algorithm used to 
track the global head motion is used, until the 
correlation function has maximized. The parameters 
describing the face model in the tracked frame are 
written into a file for later use. 



Step 5: steps 3 and 4 are repeated until the entire movie 
is processed. For best results, instead of executing this 
process serially from the first frame to the last, the 
key frames can be used as initial points of tracking. 
Every two consecutive key frames are used to track from 
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each of them to the frames between them. That 
stabilasation of the tracking is preserved. 

Step 6: The tracking described above is applied to 
dubber movie as well. 



II. NEW MOVIE generation phase 



This phase combines the tracking results of both the 
original and the dubber 's movies, in order to synthesize 
the new audio visually dubbed movie. This audio visually 
dubbed movie, as explained above, is mostly formed out of 
the original movie, except for the face of the actor in 
this audio visually dubbed movie. This face is a texture- 
mapped face on the three-dimensional face model, 
synthesized, as described above, to fit the lip, mou th, 
and cheek shapes of the dubber at that particular time. 
Thus, the parameters of the face model computed as 
described in phase I, is used to produce the new audio 
visually dubbed movie, in which for every frame in the 
original movie, the mouth parameters are modified to 
those of the dubber. The exact process is as follows: 

Step 7: For every frame in the original movie, the face 
of the actor is texture-mapped on the appropriate face 
model using the parameters that were calculated in step 3 
for the original movie. The mouth parameters of the 
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dubber as calculated in step 3 are used as follows for 
the new audio visually dubbed movie. 

Step 8: Once the desired mouth-shape of the actor is 
known, the original movie is being searched in the 
neighborhood of the current frame (approximately 0.1 - io 
seconds forwards and backwards in time) for a mouth that 
is most similar in shape or parameters to the new desired 
mouth. This search for the reference similarity frame 
takes into account the mouth-shape already chosen for the 
previous frame in order to make the mouth motion smooth 
and continuous. Prom the several (5 - io) best fit 
mouths, the mouth which is picked is from the frame that 
is closest in time to the previous picked mouth. 

Step 9: The mouth chosen in step 8 is texture-mapped into 
the mouth model using its pre-computed parameters. The 
face model parameters are then changed to the desired 
mouth shape, producing a very realistic new frame, which 
replaces the old frame in the original movie. The user of 
the program can choose the desired mouth area to be 
texture-mapped in place - it can be either the inside of 
the mouth, the whole mouth including the lips or even a 
bigger area. This procedure creates a synthesized image, 
in which the face around the mouth, and in particular the 
lips, are re-shaped according to the sound track, while 
retaining the familiar face of the original actor. step 
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8 can also be skipped, so that the inside of the mouth 
will be empty. This is useful for making a talking movie 
from still picture, where the inside information of the 
mouth is .missing; because in the reference frame 
similarity dictionary of the actor there does not exist 
any near fit of i ip shapes for the target ^ ^ 

interior can also be filled with visual color/texture. 

Step 10: Finally, the sound track from the dubbed movie 
(target text) replaces the original text and sound. 

It is to be noted that an animator using the invented 
software tool, is free to modify, set, or fix any of the 
head or mouth parameters, in both the original or audio 
visually dubbed movie, and even pick a specific mouth to 
be textured-mapped in place, as described in step NM8 , 
all of these at any of the above stages. The tracking 
program is highly interactive and user-friendly. 

The involved software (see Appendix l) of this invention 
is very versatile, and can be used in a very wide array 
of sound/text replacement applications many of which have 
been mentioned above. The following are examples of some 
applications of the present invention: 

Advertising: For products sold world-wide, an original 
advertising commercial can be manipulated to produce the 
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same commercial in any desired language. This saves the 
need to produce a new video for every country or language 
that the product is aimed for. 

Another possibility is to edit a movie, by altering 
existing scenes without having to re-shoot them again. 
If, for example, after the movie production was over, the 
director/editor wishes to alter a specific scene, or 
change one sentence of a specific actor. 

The present invention refers not only to narrated text 
but also to songs, operas, and music, opening the 
possibility to change the language of musical video 
clips . 

The production of an animated cartoon is assisted by 
drawing a line segment for the actor's mouth, drawing a 
small actor's picture dictionary containing representee 
reference similarity frames with completely drawn 
mouths, and then allowing these lip-line segments to be 
replaced with the coresponding lip shapes of the dubber 
as are to be found in the actor's picture dictionary. 

in general, applications of a method for automatic audio 
visual dubbing of movie include: cinematic movies, 
cartoons, documentaries, advertizing, news, educational 
programs, court documentations, speaches, lectures, 
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historical documentation, hearing committees, home 
videos, sports events, entertainment events, operas, 
musicals, musical video-clips, simultaneous translation, 
and adding s'peach to sequences of either original or 
added still frames of the aforesaid. 

Furthermore, using the library of the previously 
described reference similarity frames, the present 
invention allows one to create new movies altogether, and 
also to convert background narrative to audio visual 
speech, and to audio-visualize any written text. 

The present invention will be further described by 
Figures 1-4. These figures are solely intended to 
illustrate the prefered embodyment of the invention and 
are not intended to limit the scope of the invention in 
any manner. Likewise the attached software (Appendix l) 
represents an example of the implementation of the method 
disclosed in the present patent and is not intended to 
limit the scope of said method in any way. 

Figure 1 illustrates a block diagram showing the method 
stages, and its contents is is detailed below. 

Figures 2a and 2b illustrate an example of significant 
points on a generic frontal picture of a head (Figure 2a) 
and a generic side profile picture of a head (Figure 2b). 
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Figure 3 shows an example of a generic wire frame face 
model . 

For illustrative purposes one can take the significant 
points shown in Figures 2, measure them on pictures of a 
real actor, and apply them to a generic wire frame face 
model (Figure 3). Fitting a three-dimensional head model 
to the actor's two-dimensional head picture by adapting 
the data of the significant points, as measured, results 
in an integration, as can be seen in Figures 4a and 4b. 

Figure 4a is an example showing how a custom fitted wire 
frame model fits onto a frontal view of an actor's face. 

Figure 4b is an example showing how a custom fitted wire 
frame model fits onto a profile view of an actor's face. 

Figure l illustrates a block diagram showing the method 
stages : 

In the original movie (l) a frame is selected (2) having 
a nearly frontal picture of the original actor's head 
and, if available, a frame is also selected having a 
picture with his side profile. 

A three-dimensional head model is fitted to the actor's 
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description for a new picture identical to the original, 
except that the actor's mouth status resembles the mouth 
status of the dubber; wherein the new picture coresponds 
to a frame in the new audio visually dubbed movie. 

In order to overcome difficulties like for example those 
that arise when the dubber in (8) opens a closed mouth 
in (4), a frame or frames in the original movie are 
sought whose mouth status is similar to the desired new 
mouth status. The6e frames, termed reference similarity 
frames, are usually but not necessarily in a temporal 
proximity to the processed frame, and the lips from that 
frame are copied using texture mapping (12) into the lips 
area in the new frame. The search for a reference 
similarity frame is an essential component of the present 
invention. We therefore repeat its definition: a 
reference similarity frame is a picture (being a frame 
in the original movie, a frame in any other movie, or a 
still photograph) in which the original actor has the 
desired features of the mouth-shape and head posture 
suitable for the audio visually dubbed movie. 

Alternatively, the reference similarity frame may be 
taken from a compiled picture library of the original 
actor or of other actors. 
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The process (12) is repeated all over again for each 
frame, until the entire movie is converted. 
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FACE IMAGING Ltd source code for program 'dub' 

The following printout includes the source for the 
following modules: 



dub. c 
general . c 
gimg.h 

mv.h 
list .h 

io.c 

texture . c 



main program 

general functions and utilities 
interface for gimg.c - image I/O and 
filtering 

interface for mv.c - moving handling 

interface for list.c - list data type 

handling 

I/O functions 

texture mapping stuff 



These modules comprises the program 'dub' which 

automatically fixes the lips of a person in a movie to 

another person's lip shapes, as described in the 
invention description. 



The modules included are the main modules. Utility 
modules, such as handling images, movies, etc. are not 
included. 
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* Module: dub.c 

* By: Ranco 

* History: 

* 18.8.94: creation. 
••♦••••••♦•••♦•••••^♦♦••♦♦•^ 

^include <cvcnt.h> - 
^include "x_inc.h M 
^include "defs.h" 
#include "image.h" 
#include "generaLh" 
#includc "mv.h" 
#include "vars.h" 
include "dub.h" 
^include "gimg.h" 
#include "io.h" 
^include ,, texturc.h ,, 
#include <math.h> 
#include <dmedia/cl_cosmo.h> 
//include <audiofile.h> 

Movie movie t out_moviel f out_movie2,trans_rnovie; 
XtAppContext ac; 
Widget tl, da; 
Display *dpy; 
GC gc; 

char output_moviejuune[MAX_NAME], source_movie jiarae[MAXJ^AME), 

tnms_movie_name[MAXJ<AME]4u^ 

force Jrames_fileiiame[MAX_NA^ 

audio_file_name [MAX_N AME] ; 
int first frame, lastjrame, trans Iator_first_rrame; 
int sample = 1, frame jrange = FRAME_AREA, jpeg quality; 
float mouth_scale; 

int avg_params_flag = FALSE. avg_width_flag = TRUE, force_frames_flag = FALSE; 
int fulljlag = FALSE; 

Parameter **left_source_array I **right_source_array; 
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Parameter **left_trans_array, ••right_trans - axray; 
extern int texture_map; 
Gimg cuir_frame 1 ; 

int one_frame_mode, normalize_mouth, verbosemode; 

int which_params[] 

{JAW_ROTATION,Mra_Yscl,RSEJJPLff^ 

,MTH_INTERP}; 

intrangesf] = {1,2,1,1,1,0,1}; 

AFfilehandle audio 1 = AF NULL FILEHANDLE, audio2 = AF_NULL_FILEHANDLE; 

int play_flag; 

/* 

* GLX configuration parameter: 

* Double buffering 

* color index (default so unspecified) 

* nothing else special 
*/ 

static GLXconfig glxConfig Q = { 
{ GLX_NORMAL, GLX_DOUBLE, FALSE }, 
{ GLX_NORMAL, GLX_ZSIZE, 7 }, 
{ GLXJMORMAL, GLX_RGB, TRUE }, 
{ 0, 0, 0 } 

}; 



/* texture properties for the whole face */ 

float texpropsl[] = { 
TX_MINFILTER, TXJ>OINT, 
TX JvlAGFILTER, TXJ>OINT, 
TXJVRAP, TX_CLAMP, TX_NULL}; 

/* texture properties for the lips */ 

float texprops2[] = { 
TX_MINFELTER, TXBIUNEAR, 
TX MAGFILTER, TX^BILINEAR, 
TX_WRAP, TXJXAMP, TX_NULL}; 

/* Texture environment for the face and the lips */ 
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float tevpropsl[] = {TV DECAL, TV_NULL}; 

/••••••••••—••••••••• 

void Rcfresh(int dl, int d2) 
{ 

if (RcdrawNccded) { 
recompute_face(); 
DrawSccne(); 
RedrawNecdcd = FALSE; 

} 

} 



void cancclCBfWidget w, XtPointcr xtp, GlxDrawCallbackStnict *call_data) 

{ 

exit(0); 

} 

void da_initCB(Widgct w, XtPointcr xtp, GlxDrawCallbackStnict *call_data) 
{ 

zbuffer(TRUE); 
winconstraints(); 
reshape viewportO ; 
subpixel(TRUE); 



mmode(MVIEWING); 
oitho(-xsize/2.(H.5,xsize/2.(RJ,-^ 



zfar = getgdesc(GD_ZMAX); 

lsetdepth(0, zfar); 

zclcarO; 

cpack(bgc); 

clcar(); 

concavc(FALSE); 
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mmodc(MPROJECTION); 
gctmatrix(mp); 
mmodc(MVIEWING); 
frontbuffci(TRUE); 
RedrawNccdcd = TRUE; 

} 



void da_resetCB(Widget w, XtPointer xtp, GIxDrawCallbackStnict *call_data) 

{ 

RedrawNeeded = TRUE; 

} 



void UIInit(int *argc, char **argv) 
{ 

Widget cancel, form, rc,rcl, go, sep, label; 
int i; 

XColor color, uu; 
Colormap cmap; 

Pixel bg_color, top_shadow, bottom_shadow, fg, select_color, 
XmString frame_str = XmStringCreateSimple( "Frame: "), 

labcl_str = XmStringCreateSimpleC DEMO "). 

wf_str = XmStragCreateSimple(" 1/1 "), 

go_str = XmStringCreateSimpleC 1 Go!"), 

cance!_str = XinStringCreateSiniple("Quit"); 

tl = XtAppInitialize (&ac, "Dub", NULL, 0, argc, argv, NULL, NULL, 0); 
dpy = XtDisplay (tl); 

form = XtVaCreateManagedWidget("fonn", xmFormWidgetClass, tl, 
XmNnoResize, TRUE, 
NULL); 

da = XtVaCreateManagedWidget ("da", glxMDrawWidgetClass, form, 
GlxNglxConfig, glxConfig, 
XmN width, xsize, 
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XmNheight, ysize, 

XmNbottoraAttachment, XmATTACHFORM, 

XmNtopOffsct, 2, 

XmNbottomOflset, 2, 

XmNbackground, WHITE, 

NULL); 

XtAddCallback(da, GbcNresizeCallback, (XtCallbackProc)da_resetCB, NULL); 
XtAddCallback(da, GlxNexposeCallback, (XtCallbackProc)da^rcsctCB, NULL); 
XtAddCallback(da, GlxNginitCallback, (XtCallbackProc)da_initCB, NULL); 
XtRealizeWidgct (tl); 

} 



void Init(int argc, char **argv) 

{ 

char vbuf!I50]; 

long zbuf = getgdesc(GD_BrrS_NORM_ZBUFFER); 
long rgb_sbits[3], rgb_dbits[3]; 

int i, ch, source_movie_namejlag = 0, trans_movie_name_flag = 0,last_frajne_flag=0; 
int n^frames.prm; 

char model_filcnamc[MAX_NAME]; 

char temp[MAX_NAME], params[MAX_N AME] ; 

xsize = getgdesc(GD_XPMAX); 

if (argc < 5) 
ProgUsage(argv[0]); 

/* See if we're on a GT. */ 
gversion( vbuff ); 

rgb_sbits[0] = getgdesc(GD_BITS_NORM_SNG_RED); 
rgb_sbits[l] = getgdesc(GD_BITSJ*ORM_SNGJjREEN); 
rgb_sbits[2] = getgdesc(GD_BirS_NORM_SNG_BLUE); 
rgb_dbits[0] - getgdesc(GD_BITS_NORM_DBL_RED); 
rgbdbits[l] - getgdesc(GD_BITS_NORM_DBL_GREEN); 
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rgb_dbits[2] = getgdesc(GD_BITS_NORM_DBL_BLUE); 
ok_for_shading= ( 

(rgb_sbits[0]X)) && 

(rgb_sbits[l)>0) && 

(rgb_sbits[2]X))); 

ok_for_double_bufTering= ( 

(rgb_dbits[0]>=OK_FOR_DOUBLE_BUFFERING) && 
(rgb_dbits[ 1 ]>K)K_FOR_DOUBLE_BUFFERING) && 
(rgb_dbits[2]>=OK_FOR_DOUBLE_BUFFERING)); 

ifl[debug) { 

rprintf(stdout, "Machine is %s at res %ld\n ,, ,vbuff,xsize); 
rprmtf(stdout, M zbuffer bits is %ld\n M ,zbuf); 

fprintflstdouCrgb single is %l(t%ld,%ld\n",rgb_sbits[0],rgb - sbits[l],rgb_sbits[2]); 
fprintf(stdout,"rgb double is %ld,%ld,%ld^^^gb^dbitsfO] t ^gb_dbits[l] l rgb_dbits[23); 

} 

/* initialize some globals */ 
prefix = "DATA/face_model"; 
texture_map = FALSE; 
param_face = TRUE; 
eye_position_not_set = TRUE; 
show_model = TRUE; 
lips_onJy = FALSE; 
put_image = TRUE; 
which_image = FRONTIMG; 
show_teeth = FALSE; 
show_coord = TRUE; 
wire = TRUE; 
one_frame_mode = FALSE; 
moutharea = TEX_NONE; 
normalize_mouth = TRUE; 
verbosemode = FALSE; 
current_window = MAIN; 
curr_saved_image = 1; 
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RcdrawNccdcd = RccalcNeeded = TRUE; 
mouth_scale = 1.0; 

firstjrame = translator first frame = DEF_FIRST_FRAME; 

strcpy(output_movie_narne, DEF_OUTPUT_MOVIEJnJAME); 

paramsfO] = 0; 

jpeg_quality ° 93; 

playjlag = FALSE; 

bgc = 0x00202020; 

i- 1; 

tcxmode = DUBMODE; 



while(-argc && (ch ■ argv[i++][l])) { 
switch (ch) { 
case V: /* Source movie */ 
strcpy(source_movie_name, argv[i++]); 
sourcejiiovie_name_flag++; 
break; 

case V: /• New movie •/ 

strcpy(output_movie_name, argv[i++]); 

break; 
case Y: 

strcpy(trans_movie_name, axgv[i++]); 
trans_movie_naine_flag++; 
break; 
case V: 

srjcpy(force_frames_filenarne, argv[i++]); 

force_frames_flag++; 

break; 
case T: 

first_frame = atoi(argv[i++]); 

break; 
case T: 

last_frame = atoi(argv[i++]); 

last_frame_flag++; 

break; 
case *q* : 
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jpcg_quality = atoi(argv[i++]); 
sprintf(tcmp,"-q_%d n jpcg_quality); 
strcat(params,temp); 
break; 
case 'm': 

if (stincmp(argv[i-l],' , -mc",strlen( M -mc M )) = 0) { 
sscanf{argv[i++] > "%lx ,, ,&bgc); 
printfCMouth color = %x\n" f bgc); 

} 

else { 

mouth_scale = atof(argv[i-H-]); 

sprintfl[temp, ,, -m_%.2r,mouth_scale); 

strcat(params,temp); 

} 

break; 
case T': 

translator_first_frame = atoi(argv[i++)); 

break; 
case V: 

frame_range = atoi(argv[i++]); 

sprintf(ternp/ , -r_%d n t fTame_range); 

strcat(params t temp); 

break; 
case *w*: 

strcat(params, w -w M ); 

wire = TRUE; 

putjmage » FALSE; 

argc-H-; 

break; 
case 'a*: 

if (argv[i-l][2] = *p') avg_pararnsjlag = TRUE; 
else if (argv[i-l][2] = *w') avg_width_flag = TRUE; 
strcat(params, argv [i- 1 ]); 
argc++; 
break; 
case V: 
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fulljlag = TRUE; 
argc++; 

break; 
case f n*: 
normalizc_mouth = FALSE; 
argc++; 
break; 

case T: /* use INside of mouth */ 
if (argv[i-l][2] = *N') mouth_area - TEXJN; 
strcaKparams/'-IN"); 
argc++; 
break; 

case 'E': /• use EXTERNAL mouth •/ 
if (!strocmp(argv[i-l}+2, W XT,2)) mouth.area = TEX_EXT; 
strca^params^-EXT"); 
argc++; 
break; 

case 'O*: /* use OUTside of mouth */ 

if (argv[i-l][2] = 'U') mouthjirea = TEX_OUT; 

strcat(params, n -OUT'); 

argc++; 

break; 
case V: 

verbose_mode = TRUE; 

argc++; 

break; 
default: 

rprintflstderr, "%c is an illegal flgg!W, (char)ch); 
argc++; 

break; * 

> 

argc-; 
} 

if (!source_movie_name_flag |) ! trans_movie_name) ProgUsage(argv[0]); 
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make_face(); /* read data, topology and parameters. */ 
if (!put_image) { 



} 

object_is_visib!e[FLESH] = TRUE; 
object_is_visiblc[LIPS] = TRUE; 
object_is_visible[EYELASH] = TRUE; 
object Js_visible[TEETH] = FALSE; 
object_is_visible[PUPIL] = FALSE; 
object_is_visible[IRIS] = FALSE; 
object_is_visible[FRINGE] = FALSE; 
object_is_visible[EYEWHITE] - FALSE; 
setPolarAzimuth(O.O); 
setPolarInc(90.0); 
setPolarTwist(O.O); 

saved_firont_sc = saved_side_sc = sc; 

sprintf{model_filename/ , %s.mcKlel M ,soiirce_movie_iiame); 
restoreModeI(model_filename); 

sprintfOemp, ,, %s.%dprm M , source_movie_narne, first_frame); 
PRINTFlCRestoring params from %s\n", temp); 
restoreParameters(temp,leftj>aram^ 

mth_color = 01; 

/• Create output movies and open input movies */ 
if ((movie - MO VlEj>pen(source_movie_tiame)) — NULL) { 
exit(0); 

> 

xsize = MOVIE_frameWidth(movie); 
ysize = MO VTE_frameHeight(movie); 
printfCSource frame size is (%dx%d)\n",xsize,ysize); 

if ((trans_movie « MO VTE j)pen(trans_movie_name)) = NULL) { 



fgc=0xFF000000; 
bgc=OxFFFFFFFF; 



/* set background color. */ 
/* set foreground color. */ 
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exit(O); 

} 

sprintf{audio_filc_namc ( t, %s.a n ,trans_movie_name); 
if (fulljlag) { 
sprmtf(fiill_dubbed_m^ 

printfTCreating movie: %s.mv\n^mlljiubbed_movie_name); 
out_moviel = MOVIE^crcateffiil^dubbed^movie^namc^izcysizc, 

MOVIE_frameRate(trans_movie), 

MOVTE_audioRatc(trans_movie), 

DMJTOPJTO^BOTTOM, 

DM_IMAGE_JPEG, 

DMIMAGEINTERLACEDEVEN, 

jpeg_quality); 

audio 1 » MOVIE^createAudioTrackfaudio^file^name.out^movicl); 

} 

else { 

sprintf(dubbed_movie_xiame, M %s%s n 1 output_movie_name,params); 
printf("Crcating movie: %s.mv\n",dubbed_movie_name); 
out_movie2 = MOVIE_create(dubbed_moviejiame,xsize/2,ysize/2, 

MOVTE_frameRate(trans_movie), 

MOVIE_audioRate{trans_movie), 

DM.BOTTOMJTOJTOP, 

DMIMAGEMVCl , 

DMJMAGE_NONINTERLACED, 

jpeg_quality); 

audio2 - MOVIE^createAudioTrac^audio^file^name.out^moviel); 

} 

if (!last__framejlag) last_frame « MO\TE_numOfFrames(niovie)-l; 
if (MOVIE_numOfFrames(movie) = 1) one_frame_mode = TRUE; 

curr_framel = GimgCreate (xsize,ysize,GRGBA); 

n_frames = last_frame + 1; 

MALLOC(left_source_amy,n_frames,Parameter_ptr); 
MALLOC(right_source_array,n_framcs,Parameter_ptr); 
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rcstorcAllParamctcrs(sourcc_movie_namc t n_fraincs, 

left_source_amy,right_source_array) ; 

MALLOC(lcft_trans_array,n_framcs,Paramctcr_ptr); 

MALLOC(right_trans_arTay > n_frames t Parametcrj3tr); 

rcstorcAllParamcters(trans_movie_aame,n_framcs, 
left_trans_array,right_trans_aiTay); 

if (verbose_mode) { 
for (i=0; i < XtNumbcr(which_params); i++) { 
prm - which j3arams[i]; 
printf( ,, %s\tMeftj}arametcr[prm].box_label); 

} 

} 

} 

float calcParamDist (int ftame_no) 
{ 

float distance=0 ( xnin_dist = SOMEBIGNUMBER, t[5], range; 
Parameter *pji, »pJ2, *p_rl, *p_r2; 
int k; 



pjl = left_source_amy[frarne_no]; 
p_rl = right_source_array[rrarne_no]; 
p_12 = let\_parameter; 
p_r2 = right_parameter, 

range = MAXJ/AI^pJl[JAWJlOTATION]^ 

t[0] = (VAL(p^ll[JAW_ROTATION]).VAL(p^l2[JAW_ROTATION]))/range; 

range - MAX^VAL^lltMTH^YsclJJ-Mm^Al^JlfMTH^YscI]); 
t[l] = (VAl^JlfMTH.YsclJ^VAL^^fMTH.YsclJjyrange; 
t[2] = (VAL(p_rl[MTHjrscl]^^ 

range = MAX_VAL(p_l 1 [TtSEJJPLIP])-MIN_VAL(pJ 1 [RSE_UPLEP]); 
t[3] = (VALtpJltRSE.UPLIPlVVAL^^tRSE^UPLIPlJVrangc; 
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range = MAX_VAL(pJl [LWRLIPJ^ 
t[4] - (VAL(pJl[LWRLIPJTUCK^ 

for (k=0; k<5; k++) distance += fabs(t[k]); 

return (distance); 

} 



int dist_comp (const void *dl, const void *d2) 
{ 

float diff= ((distjst *)dl)->distance - ((dist_st *)d2)->distance; 
if (diff< 0.0) return -1; 
else if (diff > 0.0) return 1; 
return 0; 

} 



int angleTooFar (int a) 
{ 

Parameter *ls = lefT_source_array[a] t *lp = lef\j>arameter; 

float twist 1 = VAL(ls[POLAR_TWIST]), twist2 = VAL(lp[POLAR_TWIST]); 

if (twistl >= 180.0) twistl -= 360.0; 
if (twist2 >= 180.0) twist2 -= 360.0; 

if «fabs(VAMls[P01jflJl^AZIM]).VAL(lp[P0LAR__A2IM])) > 10.0) || 
(fabs(VAL(ls[POLAR_INC]).VAL(lp[POLAR_INC])) > 10.0) || 
(fabs(twistl-twist2) > 10.0)) { 
return (TRUE); 

} 

return (FALSE); 

} 



int scaleTooFar (int a) 
{ 
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Parameter *ls = ieft_source_array[a], *lp ■ left_parameter; 
float ratio = VAL(ls[SCALE]yVAL(rp[SCALE]); 

if (ratio < 0.9 || ratio > 1.1) return (TRUE); 
return (FALSE); 

} 



int loadMouth (int rrame_no, int lastjrhosen) 
{ 

int i, j, k, best Jit = SOME_BIG_NUMBER, widttheight^index = SOME_BIG_NUMBER, 

closest, d, found 1, bigger_raouth; 
int from = MAX(flret_frame,rramc_no-frame_rangc), 

to = MIN(last_frame f frame_no+rrame_range); 
float distance, min_dist = SOME_BIG_NUMBER; 
char temp[MAX_NAME); 
Gimg sub_img; 
dist_st *distances; 

for (i=from; i < to; i++) { 
distance = calcParamDist(i); 
distance /= fsin(RAD(1.5*(i-last_chosen)+90.0)); 
if (distance-min_dist<-0.000001) { 

min_dist = distance; 

best_fit = i; 

printf( w Frame#%d: best fit is rrame#%d (%f)\n w , rrame.ncbes^fi^min^dist); 

r 

MOVTE_getFrame (movie, best_fit,ciinjramel); 

sprint£(temp, ,, %s.%d.tex M ,source_movic_name, bestjit); 
restoreTexCoord(temp,left_mth_pts,right_mthj)ts, 

left_mtojex,right_mm_tex); /* Also sets model region vars */ 
recompute_face(); 

width = (int)(right_model-]cft__model); 
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sub_img = GimgCreatcSubCopy(cun.framel,(int)lcft_model, 

(int)bottom_modcI,width,hcight); 
tevdef{l, 0, tevpropsl); 

GhngSetAlpha(sub_img,255l); 

texdef2d(2, 4, width,height, (ujong *)ImgRast(sub_img), 7, texpropsl); 
GimgSetAlphafsubJnig^SSl); 

texdef2d(3, 4, width,height, (ujong ♦JImgRastCsubJmg), 7, tcxpropsl); 

GimgDel(sub_img); 

return (best_fit); 

} 



int insertMouth (int frame_no, int best_fit) 
{ 

int i, j, k,width,height; 
char temp[MAX_NAME]; 
Gimg sub_img; 

printf( M Frame#%d: mouth taken from frame#%d\n H , frame_no, best Jit); 

loadActorLipsParameters(best_fit); 

DrawSceneQ; 

MOVIE_getFrame (movie, bestjit,currjframel); 
sprinti^teinp > w %s.%itex\source_movie_iiame t best_fit); 
restoreTexCoord(tempJe£t - mthj)ts t right_mth_pts, 

ler\mthjex,right_mth_tex); /* Also sets model region vars */ 
recompute Jace(); 

width = (intXright_modeMeft_model); 
height = (mt)(top_model-bottom_mode]); 

tevdefl[l, 0, tevpropsl); 

sub_img =GimgCreateSubCopy(cuiT Jrame 1 ,(int)left_model. 
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(int)bottom_model,width t height); 
GimgSetAlpha(sub_img,2551); 

texdef2d(2 > 4, width, height, (ujong *)ImgRast(sub_img), 7, texpropsl); 
GimgSetAlpha(sub_img,2551); 

texde£2d(3, 4, width.height, (ujong *)ImgRast(sub_img), 7, texpropsl); 

GimgDel(sub_img); 

return (best_fit); " 

} 



void IoadParameters(Parameter *!eft_array, Parameter *right_array) 

{ 

int i; 

for (i=0; i<MAX_PARAMETERS; i++) { 
left_parameter[i] - left_array[i]; 
right_j)ararneter{i] = right_array[i]; 

} 

} 



float normalizedParam(Parameter **trans_params, 
Parameter *source_j>arams, 
int frame_no, 
int prm) 

{ 

float res, v, max_t, min_t, rain_s, max_s; 

minj = MIN^VAL(trans_j5arams[0][prm]); 

max_t = MAX^VAL(transj3arains[0][pnn]); 

min_s = MIN_VAL(source_params[pnn])*mouth_scale; 

max_s = MAX_VAL(sourcej3arams[prrn])*mouth_scale; 

v - (VAL(transj>arairis[fiaroe_no]^ 
res = v * (max_s-min_s) + min_s; 
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return (res); 



} 



void smoothLipsParameters(int firame_no, int prm, int range, float *1, float *r, 
int normalize_flag) 

{ 

float 11 = 0.0, rl - 0.0, minj = SOMEBIGNUMBER, 

mm_r - SOMEBIGNUMBER, maxj = -SOMEBIGNUMBER, i 
-SOME_BlG_NUMBER; 
float cur_I, curj; 

int i, from = MAX(fim_framc,frame_no-range), 
to = MIN(last_frame,frarne_no+range); 

if (normalize_flag) { 
curj = normalizedParam(left_trai^ 
cur_r = normalizeoTaram(rightj^ 
for (i=from-frame_no; i <= to-frame_no; i++) { 

11 += normalizedParam(leftjran^^ 

rl += nonnalizedPanmi(right_tra^ 

if (11 < minj) minj = 11; 

if (11 > maxj) maxj = 11; 

if (rl < min_r) minj = rl; 

if (rl > maxr) max_r = rl; 

} 

I 

else { 

if (prm = MTH_Yscl) { 
curj = VAL(left_trans_array[fjame - no][prrn])*mouth_scale; 
cur _ r = VAL(right_traiis_array[rraine_no][pnn])*rnouth_scale; 

} 

else { 

curj = VAUleft_traiis_array[rrame_no][pnn]); 
cur_r = VALtright^trans^an-aylframe^oltprm]); 

} 

for (i=*rom-frame_no; i <= to-frame_no; i++) { 
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if (prm = MTH_Yscl) { 
U += VAL(icft_trans_arTay[firame_no+i][prm])*mouth_scalc; 
rl += VAL(right_trans_array[frame_no+i][pnn])*mouth_scale; 

} 

else { 

11 += VAL(left_trans_array[frame_no+i][pnn]); 
rl += VAL(right_trans_amy[tiame_no+i][prm]); 

} 

if (11 < min J) min_l = 11; 
if (11 > max J) max J = U; 
if (rl < min_r) min_r = rl; 
if (rl > max_r) max r - rl; 

} 

" > 

♦1 = 11 /(to-from+1); 
*r = rl /(to-from+1); 

} 



void loadTranslatorLipsParameters(int framejio, int normalize_params) 
{ 

float Il,12,l3,rl,r2,r3; 
int i, prm; 

int from = MAX(first_framc,rramc_no-frarne_range), 
to = MIN(last_rrame t rrame_no+frame_range); 



for (i=0; i < XtNumber(which_params); i++) { 
prm = which_params[i]; 



if (avg_params_flag) 

smoothLipsParameters(frame_no,prm, ranges [i],&12, &r2, normalize jjarams); 
else 

smoothLipsParametere(frame_no,prm, 0, &12, &r2, normaIize_params); 
if (prm = MTH_Yscl || prm = MTH_INTERP) { 



SUBSTITUTE SHEET (RULE 26) 



WO 97/15926 PCT/1B96/01056 

46 

if (avg_width_flag) 
VAL(lcftj3aramctcr[pnn]) = VAL(rightjjaramctcr[pnn]) = (12+r2)/2.0; 

} 

else { 

VAL(leftj)arameter[pnn]) =12; 
VAL(right_j>arameter[pnn]) = r2; 

} 

PRimT2(«changed to:(%f t o/ o0 \ n -,VAL(Ieft^ 
if (verbose_mode) printfl[ <> (%.4f.K.40\tM2 v i2); 

> 

if (verbose_mode) printfCV* 1 ); 
recompute jFace(); 

} 



void loadActorLipsParameters(int frame_no) 
{ 

int i, pnn; 

float Il,12 t 13,rl t r2,r3; 

for (i=0; i < XtNumber( which j>arams); i++) { 
prm « which_params[i]; 

VAL(left_parameter[pmi]) = VAmeft^source^arraytframe^nolIpnii]); 
VAL(rightjarameter[prm]) - VAL(righLsource^array[frarne_no][prm]);; 

} 

recompute_face(); 

} 



void Go(void) 
{ 

long nof; /• Num of frames */ 

char temp[MAX_NAME], line[80]; 

int i, j, last, orig_frame_no,new_frarne_no; 



7 
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Gimg small_img, grab_frame; 
time_t tm; 

float p!6[2],p288[2], distance.distancel.sb; 
Matrix M; 

FILE *force_frames; 

int compressed_image_size, •compressed_image; 

if (force_frames_flag) { 
if ((forcejrames = fopen(forceJramesJi!ename, V)) = NULL) 

forceJrame$_flag = FALSE; 
else { 

fgets (line, 80, forcejrames); 

sscanflline, "%dto%d'\ &orig_frame_no, &new_frame_no); 

} 

} 

tm = time(O); 

printfTStart dubbing %d frames at %s^astjrame-firstjrame, ctime(&tm)); 

readsource(SRC_FRONT); 

grab_frame - GimgCreate(xsize,ysize,GRGBA); 

MOVTE_setCuiTcntFrameNo (movie, first_frame); 
if (one_frame_mode) { 

textureMapping(TEX_ON); 

DrawScene(); 

} 

cpackfbgc); 

for (i=first_frame, j==translator_first_rrame; i <= last_frame; i++j++) { 
tex_mode = TRACK_MODE; 
sprintf(temp, "%s.%dprm", source_movie_name > i); 

if (lone frame_mode) loadParameters(left - source_amy[i],right_source_array[i]); 
/* else loadTarametentfleft_sourcejim^ 
else { 

VAL(lcft_parameter[BEARD^SCL]) = VAL(right_paramcter[BEARD^SCL]) = 0; 
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recompute_face(); 

} 

VAL(ler\j)arameter[BEARD_SCL]) = VAL(right_paramcter[BEARD_SCL]) = 0; 

recompute_face(); 

DrawSccne(); 

prepareMatrix(M); 

if (put_image && lone frame_mode) { 
textureMapping(TEX_ON); 
DrawScene(); 

) 

loadTranslatorLipsParametcrs(i,normalize_mouth); 
DrawSccncO; 

calcProjection(left_trans_pts,right_trans_pts); 
if (mouth_area >= TEX JN) { 
if (force Jrames_flag && (i — orig_frame_no)) { 
last = inscrtMouth(orig_ftame_no,ncw_framc_no); 
fgets (line, 80, force_frames); 

sscanf(iine, M %d\t%d'\ &orig_frame__no, &new_frame_no); 

> 

else { 
if (i = iirst_rrame) 

last = loadMouth(i,i); 
else 

last = loadMouth(i,last); 

} 

> 

else 

printf( n Franie#%d\n w , i); 

tex_mode = DUB_MODE; 
recompute_face(); 
prepareMatrix(M); 
DrawScene(); 
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lrec tread(0,O,xsize- 1 .ysize- 1 ,(u Jong *)ImgRast(grab_frame)); 
MOVIE_addCurrentFraine(out jnovie 1 ,grab_frame,0); 
MO VlE_addAudioFrame(audio 1 ,out_movie 1 ); 
} 

else { 

small_img = GimgResize(grabJrame,xsize/2,ysize/2,2); 
MOVIE_addCunentFi^e(out_movie2 t snial]_img,0); 
MOVIE_addAudioFrame(audio2,out_movie2); 
GimgFree(small_inig); 

} 

if (put_image && ! oneframemode) { 
MO VIE_nextFrame(movie) ; 
textureMapping(TEX_OFF); 

} 

} 

if (folljlag) { 
MOVIE_close(out_movie 1 ); 

out_moviel = MOVIE_open(fUljiubbed jnovie jiame); 

} 

else { 

MOVIE_close(out_movie2); 

out_movie2 = MOVIE_open(dubbed_movie_name); 

} 

tm = tirae(O); 
AFcIosefile(audiol); 
AFclosefile(audio2); 
printf(Tinished at %s w , ctime(&tm)); 

} 

/...•..•••••••..•.•.••••^^ 

void DrawScene(void) 

{ 

zclear<); 

update_window_display(); 

} 
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void ProgUsage(char *filename) 

{ 

fprintflstderr/Dub: fa lips in a dubbed movic.\n n ); 

fprintHstderr.'OJsagc-.Vii"); 

fprintf(stderr t " %s [options]\n*\ filename); 

fprintf(stderr f "\n Options:\n H ); 

fprintflsldeir," -s <movie file>\tSource: movie to be dubbed\n"); 
fprint^stderr," -o <movie file>\tOutput movie name (default s =\"movieV , )\n"); 
fprintf(stderr > M -t <movie file>\tTranslator's movie\n M ); 

fprintf][stden\" -u <filenamc>\t\tForce usage of info in filename to plant mouth\n"); 

fprintflstdenV' -f <intcger>\t\tFirst frame num, starting at 0 (default)\n M ); 

rprintf^stde^r t ,, -1 <integer>\t\tLast frame num (default=0)\n"); 

fprintflstderr," -F <integer>\t\tTranslator*s First frame num (default=0)\n M ); 

fprintf(stderT," -q <integer>\t\tJpeg quality (default=93)\n ,t ); 

fprintflstderr," -m <integer>\t\tMouth scale factor (optional)\n"); 

fprintflstderr," -r <integer>\t\tRange of frame to search mouth (defalut=25)\n n ); 

fp^intf^stde^T l ,, -w \t\t\tCreate only Wire frame animation (opnonal)\n M ); 

fprintflstderr," -ap \t\t\tAverage params (optional)\n M ); 

fprintfCstderr," -aw \t\t\tAverage mouth width (optional)\n"); 

fprintflstderr," -b \t\t\tCreate a full size movie (optional)\n M ); 

fprintffstden," -n \t\t\tDon*t Normalize mouth\n M ); 

fprintf(stderr, M -IN \t\t\tUse inside of mouth (default)\n H ); 

fprintflstdeTT," -OUT \t\t\tUse outside of mouth\n"); 

fprimffstderr/' -EXT \t\t\tUse bigger area around mouth\n B ); 

fprintflstderT t " -mc \t\t\tlnside mouth color\n"); 

fprintflstderr," - v \t\t\tVerbose mode\n M ); 

exit(O); 

) 

main(int argc, char **argv) 
{ 

XEvent e; 
char temp[200]; 
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Init(argc, argv); 

UIInit(&argc, argv); 

draw_poly = 1; 

MOVIE_bind(movie, da); 

MOVIE_setCuiTcntFranieNo(movic,first_frame); 

draw_face(); 

RedrawNeeded = TRUE; 
while (XtAppPending(ac)){ 

XtAppNcxtEvcnt (ac, &e); 

XtDispatchEvcnt (&e); 

} 

Go(); 

printfrDonelNn"); 

) 

* Module: general.c 

* By: Ranco 

* History: 

* 18.8.94: creation. 

#include M x_inc.h" 
#include "mv.h" 
tfinclude "general.h" 

Set the proper GL matrices: The transformation that a point p experience 
is as follows: 

p • (S * Rx(-90)*Ry(.90)*Rx(90-x)*Ry(-y)R2) • TRANS * Tnew = (U\ V) 

void setMatrices(void) 
{ 

loadmatrix(idmat); 

translate (left^arameterfHEADJr'ofr).vaiue 1 
left^parameterpiEAD^ZoiTl.value, 
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left_paramctcrfHEAD_Xof!].vaiuc); 
rotatcAndScalcO; 

} 



void rotateAndScale(void) 
< 

/* Global transformation: */ 
/* multnntrix(global_trans);*/ 

rotatcttimJCleftjjarameterfPOLAR^TWIST]. value* 1 0),V); 
rotatc(-(int)(lcft _parametcr[POLAR_AZIM]. value* 10), 'y 

^otatc(90(Hint)(left_pa^arneter[POLAR - INC).value*10)/x , ); 
rotate(-900 f y); 

rotate(-900,V); 

if (whichjmage = SIDEJMG) 

scale(saved_sidelsc,saved_side_sc,saved_side_sc); 
else 

scale(sc,sc,sc); 

} 



void get_composite_matrix(Matrix M) 
{ 

Matrix mv; 
register int ij,k; 

mmode(MVIEWING); 
getmatrix(mv); 

/* form product mv * mp .... viewing followed by projection */ 
for (i=0; i<4; i++) { 
for (k=0; k<4; k++) { 
M[i][k] = 0.0; 

for 0=0; j<4; j++) M[i][k] += mv[i][j] * mp[i][k]; 

} 
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} 

} 



I* Tipically, both functions above should be called together, and this is */ 
/* what this function does... */ 



void prepareMatrix(Matrix M) 
{ 

sctMatricesO; 

get_composite_matrix(M); /* View x Projection matrix */ 



/***•**•««***** World to screen coordinates **************»*********•**•****/ 

void world2screen(Matrix M, float x, float y, float z, float *sx, float *sy) 
{ 

float v[4); 
register int i; 

long xs - xsize, ys = ysize; 

for (i=0; i<4; i++) v[i] = x*M[0][i]+y*M[l][i]+z*M[2][i]+M[3][i]; 

if (v[3J=0.0) { 
Bug("w2s: v[3] = 0.0 H ); 

fprintf(stderr, M V (%f, %f, %f, %f)\n", v[0] f v[l], v[2], v[3]); 

return; 

} 

*sx = ((float)xs/2.0)*(v[0]+1.0)-.5; 
*sy = ((float)ys/2.G)*(v[l]+1.0)-.5; 

} 



/•*•*******♦• Returns the distance between 2 points ***♦***♦****•*♦*♦•**••**/ 

float dist (float *pntl, float *pnt2) 

{ 

return (fsqrt((pntl[0]-pnt2[0]) * (pntl[0]-pnt2[0]) + 
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(pntl[l]-pnt2[l]) • (pntl[l]-pnt2[l]))); 

} 

float hdist (float *pntl, float *pnt2) 

{ 

return (fabs(pntl[0]-pnt2[0])); 

} 

float vdist (float *pntl, float *pnt2) 
{ 

return (fabs(pntl[l]-pnt2[l])); 

) 



/* Matrix utilities */ 

/*«*»»******«*»********•***•,*•„„„ 

void copy_matrix (Matrix to, Matrix from) 
{ 

register int i, j; 

for (i=0; i<4; i-H-) 
for 0=0; j<4; 
to[i]D] - from[i]D]; 

} 



void print_matrix(char *str, Matrix m) 
{ 

register int i, j; 

printfTPrint matrix: %s\n'\ str); 
for (i=0; i<4; i++) { 
for 0=0; j<4; j++) { 
prmtfT%3.5f \m[i][j]); 

} 

printfC^n"); 

} 
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' •/ 

void setPolarTwist(float va!) 

{ 

leftjjaiamcterfPOLAR^TWISTl.value = 
right j>aramctcr[POLAR_TWIST].vaIue = val; 

} 

void setPolarAzimuth(float val) 
{ 

leftjarameter[POLAR_AZIM].value = 
right_parameter[POLAR__A2IM).value = val; 

> 



void setPolarlnc(float val) 
{ 

left_parametcr(POLAR_INCJ.vaIue = 
right j)arameter[POLARJNC].value = val; 

} 

void addDist2Face(float delta) 

{ 

sc += delta; 

! 



void addXoffffloat delta) 

{ 

left_parameter[HEAD_Xoff].value = 
right j>arameter[HEAD_XofT].vaIue += delta; 

left_parameter[HEAD_Xoff].changed = 
right j3arameter[HEAD_Xoff].changed = 1; 

} 



V 
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void addYoffffloat delta) 
{ 

leftj>arameter[HEAD_Yoff].vahie = 
right j>arameter[HEAD_Yoff].value += delta; 

leftj3arameter[HEAD_Yoff]xhanged = 
right_parameter[HEAD_Yof!].changed = 1; 

} 



void addZofflffloat delta) 
{ 

left_parameter[HEAD_Zofl].value = 
right_j5axameter[HEAD_ZofTJ.value += delta; 

left_parameter[HEAD_Zoff). changed = 
right jarameter[HEAD_Zoff]xhanged = 1; 

} 



void addPolarTwist(float delta) 
< 

leftj)arameter[POLAR^TWIST].value = right j>arameter[POLAR^TWIST]. value += delta; 
leftjarametertPOLAR^TWISTJ.changed = right j)arameter[POLAR_TWIST]. changed = 1; 



void addPolarAzimuth(float delta) 
{ 

leftj>arameter[POLAR_AZIM].value = right_parameter[POLAR^AZIM].vaiue += delta; 
leftj)arameter[POLAR_AZIM].changed = right_parameter[POLAR_AZIM]. changed = 1; 

} 



void addPolarInc(float delta) 
{ 

left_parameter[POLAR_INC].value - right j>aimeter[POLARJNC].value +« delta; 
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leftj5arametcr[POLAR_INC].changcd = right_parametcrfPOLAR_INC].changcd =1; 

} 



void saveTexCoord(char *filename, int lip) 
{ 

FILE *f; 

char temp[MAX_NAME], parain_filc_namc[MAX_NAME]; 
int i; 

Matrix M; 

float p[2],width,height; 

calcTextureCoordsflefttex^rightJex); 

sprintfl[temp, H %s.tex M f fi]cname); 

unlink(temp); 

if ((f = fopen(temp, "w")) == NULL) { 
rprintflstderr, "Can't open texture file '%s* for writing!", temp); 
return; 

} 

rprintflft "Num of points: %d\n",np); 

rprint^f,"Region: %f, %f, %f, %f\n", left_modeI, top_model, 

right_model, bottom_model); 
width = (floor)(right_model-Jert__modeI); 
height - (floor)(top_model-bottom_model); 
prepareMatrix(M); 
for(i=l; i<=np; i-H-) { 

world2screen(M, leiVpt[i][0], lefl_j>t[i][ 1 ], left_pt[i][2], &p[0], &p[l]); 

p[0] = (p[0]4eft_model)/width; 

PHI ■ (p[l]-bottom__model)/height; 

rprintiTf, "%d\t%f, %f\n n ,i,p[0] 1 p[l]); 

world2screen(M, right_pt[i][0], right_pt[i][l], right_pt[i][2], &p[0], &p[l]) ; 
p[0] = (p[0]-leftjnodel)/width; 
p[l] = (p[l]-bottom_model)/height; 
fprintiTf, "%d\t%f, °/of\n'\i,p[0],p[l]); 

} 

setPolarlnc(90.0); 
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setPolarTwist(O.O); 
setPolarAzimuth(0.0); 

sc = 0.35; /• Usually it is around this number... ♦/ 
prepareMatrix(M); 

fprintf(f, "# Normalized projection of selected points of modeI:\n n ); 
for(i=49; i<=70; i++) { 

world2screen(M, left_pt[i][0], left_pt[i][l], left_pt[i][2J. &p[0], &p[l]) : 

fprintfl.f, "%d\t%d, %d\n",i,(int)p[0], (int)p[l]) ; 

world2screen{M, right_pt[i][0]. right_pt[i][l], right_pt[i][2], &p[0], &p[l]) ; 
fprintfff, "%d\t%d, %d\n",i.(int)pfO], (inflpfl]); 

) 

i = 287; 

world2screen(M. left_pt[i][0], left_pt[i][l], left_pt[i][2], &p[0], &p[l]) ; 
fprintfl[f, "%d\t%.0f, %0f\n",i,p[0]+.5, p[l]+.5); 

world2screen(M, right_pt[i][0]. right_pt[i][l], right_pt[i][2], &p[0], &p[l]) ; 
fprintf{f, "%d\t%0f, %Of\n",i,p[0]+.5, p[l]+.5); 

fclose(f); 

s p r i n t f ( p a r a m _ f i 1 e _ n a m e . " % s . % d . p r m ■ , 
out_base_name,(int)MOVIE_getCurrentFrameNo(movie)); 
restoreParameters(param_file_name,left_parameter,right_parameter,TRUE); 

) 



void restoreTexCoord(char •filename, 

float left_pt[][2]. float right_ptrj[2], 
float left_tex[][2], float right_tex[][2]) 

< 

FILE *f; 

char temp[MAX_NAME],line[MAX_LINE] ; 
int i, j, np; 

float width,height; 

if ((f = fopen( filename, V)) = NULL) { 
sprintf(temp,"%s.tex ,, ,filename); 
if ((f = fopen(temp, V')) = NULL) { 
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fprintflstderr, "Can't open texture file *%s* for reading!", filename); 
return; 

} 

> 

fgets(line,MAX_LINE,f); 

sscanpne, "Num of points: %d\n",&np); 

fgets(lme,MAXJJNE,f); 

sscanflline, "Region: %f, %f, %f t %f\n", 

&left_model, &top_model, Aright^model, &bottom_model); 

width = (floor)(right_model-left_model); 
height = (floor)(topjnodeNbottom_model); 
for(i=l; i<=np; i++) { 

fgets{line,MAX_LINE,f); 

sscanfiMine, "%d\t%f, %r t &j t &leftjex[i][0],&lefijex[i][l]); 
fgets(line,MAX_LINE,f); 

sscanf^line, "%d\t%f, %r,&j,&rightJex[i][0] t &rightjex[i][l]) ; 

> 

fgets(line,MAX_LINE,f); /• Remark line */ 
while (!feofff)) { 

fgets(line,MAX_LINE,f); 

sscanf(line f "%d",&j); 

sscanf(line, "%*d\t%f, %f\Sdcftjptti][0]Mefi_ptU][l])l 
fgets(iine,MAX_LINE,f); 

sscaniUine, M %*d\t%f, Vof'.&right^tUlIOl^rightj)^]!!]); 

} 

fclose(f); 

} 



/* Filename should be added ".mthl" and ".mth2" */ 
/* Then, inside of mouth (without lips) coordinates are saved in .mthl */ 
/* file, and outside of the moith is saved in .mth2 file */ 
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void saveInMth(char •filename) 
{ 

FILE "i; 

char temp[MAX_NAME]; 

/* Save inside of mouth •/ 
sprintfltemp, "%s.mml", filename); 
unlink(temp); 

if ((f = fopen(temp, "w")) = NULL) { 
fprintflstderr. "Can't open mth param file '%&' for writing!", temp); 
return; 

} 

forintfff, "%f, %f\n",right_tex[287][0], right_tex[287][l]) ; /• 0 •/ 
forintflf. "%f, %f\n",right_tex[63][0), right_tex[63][l]); /• 1 ♦/ 
fprintfff, "%f, %l\n",right_tex[62][0] > right_tex[62][l]); /• 2 */ 
fprintfft "%f, %f\n",right_tex[61][0j, right_tex(61][l]); /• 3 •/ 
fprintf(f, "%f, %An",left_tex[62][0] ( left_tex[62][l]); /• 4 */ 
fprintfl[f, "%f, %f\n",left_tex[63][0], left_tex[63][l]); /• 5 •/ 
fprintftf "%f, %An",leftjex[287][0], left_tex[287][l]) ; /• 6 */ 
fprintftf "%f, %f\n",left_tex[60][0], left_tex[60][l]); I* 7 •/ 
forintftf, "%f, %An",left_tex[59][0], left_tex[59][lj); I* 8 •/ 
fprintftf "%f, %An",left_tex[58][0], left_tex[58J[l]); /• 9 •/ 
forintflf, "%f, %f\n",right_tex[59][0], right_tex[59][l]); /* 10 •/ 
fprintfff, "%f, %f\n",right_tex[60][0], right_tex[60][l]) ; /• 11 •/ 
fclose(f); 

/* Save outside of mouth ♦/ 
sprintf(temp, "%s.mth2", filename); 
unlink(temp); 

if ((f = fopen(temp, "w")) = NULL) { 
fprintfl[stderr, "Can't open mth param file '%s' for writing!", temp); 
return; 

} 

forintf(f. "%f, %fin"jight_tex[70][0], right_tex[70][l)); /• 0 */ 
fprintfTf. »o/ 0 f, %f\n",right_tex[69][0], right_tex[69][l]); /• 1 */ 
forintf{f, -/ 0 f, %fvn".right_tex[68)t0], right_tex[68][l)); /* 2 */ 
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fprintftf "%f, %f\n n ,right_tex[67][0], right_tex[67][l]) ; /♦ 3 */ 
fprintftt "%f, %f\nMeftjex[68][0], left_tex[68][l]); /* 4 •/ 
fprintftt "%f, %AnMcftjcx[69][0], left_tex[69][l]); /• 5 •/ 
fprintftf "%f, %f\n , \!eftjex[70][0] > left_tex[70][l]); /* 6 */ 
fprimflf, "%f, %iW\let\tex[51][0], leftjex[51 ][!]); /* 7 •/ 
rprintf(f, "%f, %f\n"Jcftjcx[50][0], lefi_tex[50]P]); /* 8 */ 
fprintftf "%f, %f\nMcftjcx[49][0], left_tex[49][l]); /♦ 9 */ 
fprintftf "%l %f\n\right_tex[50][0), right jcx[50][l]); /• 10 */ 
fprintftf "%f, %f\n",righMcx[51][0], right_tex[51][l]); /• 11 */ 
fclose(f); 

/* Save external mouth */ 
sprintf(temp f M %s.mth3", filename); 
unlink(temp); 

if ((f = fopen(temp, "w")) = NULL) { 
fprintflstderr, "Can't open mth paiam file '%s' for writing!", temp); 
return; 

} 

fprintftf H %f, %f\n B ,rightJex[44][0], right_tex[44][l]); /* 0 */ 
rprintftf "%f, %An",rightJex[73][0], right_tex[73][l]); /* 1 */ 
tprintftf "%£ %f\n M ,right_tex[72][0], right_tex[72][l]); /♦ 2 •/ 
fprintftf "%£ %f\n w ,rightjex[71][0], right Jex[71][l]); /* 3 */ 
fprintftf w %f, %f\nMeftjex[72JtO], leftjex[72][l]); /♦ 4 */ 
tprintflff, "%f, %f\n w ,leftjex[73][0], leftjex[73][l]); /♦ 5 •/ 
rprintf(f, "%f, %f\n",left_tex[44][0], left_tex[44][l]); /* 6 */ 
fprintf{f, "%f, %f\n",left_tex[43][0], left_tex[43][l]); /• 7 •/ 
fprintftf "%f, %An",left_tex[42][0] > left_tex[42][l]); /* 8 •/ 
fprintf(f, "%f, %f\n",left_tex[41][0], left_tex[41][i]); /• 9 •/ 
fprintftf "%f, %f\n",right_tex[42][0), right_tex[42][l]); /* 10 •/ 
fprintlTf, "%f t %f\n",rightjex[43][0], right_tex[43][l]); /♦ 11 •/ 
fclose(f); 
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int intcomp (const void const void *i2) 
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{ 

return ('((int *)il) - '((int *)i2)); 

} 

int rcadKeyFramelndiccs (char ♦filename, int *keyjrames, int ♦mitjrames) 

FILE *f; 
int i = 1 ; 

char temp[MAX_LINE] ; 

key_frames[0] = init_frames[0] ° 0; 
sprintf(temp/TQS.kf\filename); 
if ((f - fopen(temp, V)) = NULL) { 
return (FALSE); 

} 

while (fgets(temp,MAX_LINE,0 && !feof(f)) { 
sscanfltemp, M %d", &key_frames[i]); 
sscanfl[temp, "%d", &mitJrarnes[i-H-]); 

> 

fclose(f); 

key_frames[0] = i-1; 

qsort (key,frames+l,key_rTames[0],sizeof(int).int - comp); 

sprintf(tcmp/Tos.if\filename); 
if ((f - fopen(temp, V)) = NULL) { 
return (FALSE); 

) 

while (fgets(temp,MAX_LINE,f) && !feof(f)) { 
sscanf(temp f "%d H , &imtjrames[i++]); 

} 

fclose(f); 

init_rrames[0] ■ i-1; 
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qsort (init_framcs+l f init_franies[0],sizcof(iiit) f int_coinp); 
primfCftd init frames: w ,init_frames[0]); 
for (i=l; i <= init_framcs[0]; 

printf ( n %d init_firames[i]); 
printfCtaftd key frames: *\key_fiames[0]); 
for (M; i <= key_frames[0]; i++) 

printf ("%d key^frames[i]); 
p^intf^\n ,, ); 
return (TRUE); 

} 

/**•*•** 

void restoreInMth(char *filename, float into[12][2]) 

{ 

FILE *f; 
int i; 

char temp[MAX_LINE]; 

if ((f = fopen(filename, V)) = NULL) { 
return; 

} 

for (i=0; i < 12; i++) { 
fgets(temp,MAX_LINE,f); 
sscantTtemp, "%f f %f, &into[i][0], &into[i][l]); 

. } 
fclose(f); 

} 




void swapParameters(void) 



int j; 



if (which_image = SIDE_IMG) { 
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for 0=0; j < NUM SAVED; { 
saved Jront^aiam$[0][j] = VAL(!cftj)arametcr[savc^those[j]]); 
saved Jront_j>araira[l]|j] - VAL(right_paramctcr[savc_thosctj]]) ; 
VALOcft^aramctcrfsavc^thoselj]]) - saved^side_params[0][j]; 
VAL(right_paramctcr[save_thosc[j]]) = saved_sidejarams[l]fj); 

) 

} 

else { 

for 0=0; j < NUM_SAVED; j++) { 
savcd^sidej>arams[O]0] = VAL(Ieftj)arameter[save_lhose[j]]); 
savedj»dejararns[l][j] = VAL(right_parameter(save^those[j]]); 
VAL(Ieft J >arameter[save_those[fl]) = saved_fronu>arams[0][j]; 
VAL(rightjparameter[savejhose[j]]) = saved^front_params[l][j]; 

> 

} 

) 



/* Converts Gimg (unsigned long (RGB)) image to char •result, which is */ 
/* The weighted average of the input image. Expects result memory to •/ 
/* be allocated. •/ 



void longImg2charVec (ujong *irog, int sx, int sy, int width,int height, int xsize, char 'result) 

{ 

register int y, x, dx = xsize - width; 
img += sy*xsize + sx; 

for (y=0; y < height; y++) { 
for (x=0; x < width; x++) { 
♦result = (char)greyColor(*img); 
result++; 
img++ ; 

} 

img += dx; 
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void findRegion(int from_ vertex, 
int to_vertex t 
float 'left, 
float *top, . 
float *right, 
float *bottom, 
float w_scale_factor, 
float h_sca!e_factor) 

{ 

float *u, *d, % *r; 
Matrix M; 
float temp; 

float max_x, max_y, min_x, min_y; 

int size, corj, maxxi, max_y_i,min_x_i,min_yj; 

float width, height,lp[2],rp[2]; 

max_x = -99999.0, max_y = -99999.9; 
min_x = 99999.0, min_y « 99999.9; 
prepareMatrix(M); 

for (j^om^vertex; j <= to_vertex; j++) { 
world2screen(M, leftj>t[jP], !eftj>t[j][l), left_ptG][2], &lp[0], &lp[l]); 
world2screen(M, right_pt[j][0], rightj>t[fl[l], right jt[j][2], &rp[0], &rp[l]) ; 
if (GT(Ip[0] f max_x)) { max_x - lp[0]; 1 = leftjtjj^max^i^j;} 
if (GT(rp[0], max_x)) { max_x - rp[0); 1 - right_pt|j];max_x_i=j;} 
if (LT(Ip[0] t min_x)) { min_x = lp[0]; r = icftj)t[j];min_xj^j;} 
if (LT(rp[0], min_x)) { min_x = rp[0]; r = rightjtlj];minjcj=j;} 

if (GT(ip[l], max_y)) { max_y = lp[l]; u = leftjtO];max_y_H;} 
if (GT(rp[l], max_y)) { max_y = rp[l] ; u = rightjtUJ^nax^yJ^j;} 
if (LT(lp(l], min_y)) { min^y « lp[l] ; d ~ left_pt[j] ;min_y_i=j;} 
if (LT(rp[l], min_y)) { min_y = rp[l] ; d = right_pttj];miii_y_i=j;} 

} 
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world2screen(M, r[0], r[I], r[2], left, Atemp); 
world2screen(M, u[0], u[l), u[2], &temp, top); 
world2screen(M, d[0], d[l], d[2], &temp, bottom); 
world2screen(M, 1[0], 1[1], 1[2], right, &temp); 

width = (♦right) - (•left) + 1; 
height = Ctop) - (♦bottom) + 1; 
•left -= width * w_scale_factor; 
♦right += width * w_sca!e_factor; 
♦top += height * h_scale_factor; 
♦bottom -= height * h_scale_factor, 

♦left =MAX(l,(*left)); 

♦right -MINCMOVIE^frameWid^movieM, (♦right)); 
♦top = MIN(MOVIE_fTameHeight(movie)- l,(*top)); 
♦bottom - MAX(1, (♦bottom)); 

> 

void setModelRegion(void) 
{ 

fiadRegion(l,np, 

&left_model, &top_model t 
&right_model, &bottoin_model, 
0.0,0.0); 

Ieft_modei = floor(left_model); 
right_model = floor(right_model+.5); 
bottom_modeI = floor(bottom_model); 
top_model = floor(top_model+.5); 

} 

Parameter temp_left[MAX_PARAMFTERS], temp_right[MAX_PARAMETERS]; 
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int restoreLipsParameters(int ftame_no) 
{ 

int i, pun; 

float Il,12,13 t rl,r2,r3; 
char temp[MAX_N AME] ; 

int whichQ = {JAWJlOTATIONXmjV'scU^ 

sprintf(temp ) no /os.%d.pnn ,, t out_base_name, firame_no); 
if (!restorcParametcrs(temp,tcmpJcft, ternp_right,TRUE)) return (FALSE); 
for (i=0; i < XtNumbcr(which); { 
pnn = which[i]; 

VAL(left_paramcter[prm]) = VAL(tcmp_lcft[pnn]); 
VAL(right_parameter[prm]) = VAUtempjight[pnn]); 

> 

rccompute_face(); 
return (TRUE); 



void restoreModel(char *filename) 
{ 

char temp[MAX_LINE] ; . 
FILE *f; 
int i; 
float *p; 

if ((f = fopen(fllename, V)) = NULL) { 
printfT"%s not found!\n rt ,filename); 
return; 

} 

fgets (temp, MAX LINE, f); 
for (i=l; i <= np; i++) { 

fgets (temp, MAX_LINE, f); 

p = ieft_inpts[i]; 

sscanf^temp, M %*d\t%f\t%f\t%f\ &p[0], &p[l], &p[2]); 
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fgets (temp, MAX_LINE, £); 
p = right_mpts[i]; 

sscanf(tcmp, "%*d\t%At%f\t%r, &p[0] t &p[l], & p [2]) ; 

} 

fclosc(f); 

} 

••*••••••••••••••/ 

void rcstorcAllParameters(char *filename, 
int n_frames, 
Parameter **left_array, 
Parameter **right_array) 

{ 

int i, j.prm; 

char temp[MAX_NAME]; 

for (i=0; i<n_rrames; i++) { 

MALLOC(Ieft_amy[i],MAX^PAIUMETERSJ>arameter); 

MALLOC(right - airay[i],MAX_PARAMETERS,Parameter); 

bzero<Ieft_array[i],MAX^ 

bzeroCright.arraytiJ.M^ 

sprint£(temp, "%s.%±pxm\ filename, i); 

restoreParametersftempJef^arraytil.righLarraytiJJALSE); 

if (i=0 && (n_frames > 1)) { 
for (j=0; j < NUM_ACTIVEJvITH_PARAMS; j++) { 
prm = active_mth_parains[j]; 

MAX_VAL(lert^array[0][pnn]) = VAL(!eft_array[0][prm]); 
Mm_VAL(left_amy[0][prm]) = VAL(left_airay[0][prm]); 
MAX_VAL(right_array[0][prm]) - VAL(righMiiTay[0][prm]); 
MIN_VAL(righLarray[0][prm]) = VAL(rightjirray[0][prm]); 

} 
} 

else for G~0; j< NUM ACTWE MTH PARAMS; { 
prm = active_mth_params[j]; 
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iffGTfVALOcft.aiTayriJfpnnJXMAX.VAmeft.anBylOJIpnnJ))) 

MAX.VAmeftanayfOJfpnn]) = VAmeft.anBy[j][pnnJ); 
if (LT(VAL(left_array[i][pnn]), MIN.VAmeft.airayfOJIpnn]))) 

Mm_VAmeft_anay[0][pnn]) = VALdeft.anayfi]^]); 
if (GTCVAUright.anayfiJtpnnJ), MAX.VAUright.anayfOJIpnn]))) 

MAX.VAMright.a^yfOJfpnn]) = VA^ght.arTayfijrpnn]); 
if (LT(VAL(rightjmay[i][prrn]), MIN_VAMright_ana y [0]Ipnn]))i 

MIN_VAL(ri6ht_aTray[0] [ pnn]) = VALtfghtjUMyfltpnnJ); 



} 

} 



void caIcProjection(float left_pts[][2], float rightjtsfP]) 
{ 

register int j; 

float x, y. osc= sc. oinc = VAL(left_parameter[POLAR_rNC]), 
otwist = VAL(left_parameterrpOLAR_TWIST]), 
oazim - VAL(left_parameter[POLAR_AZIM]); 

Matrix M; 

setPolarInc(90.0); 

setPolarTwist(0.0); 

setPolarAziniuth(O.O); 

sc = 0.35; /• Usually it is around this number... •/ 
recompute face(); 
prepareMatrix(M); 
for(j=l;j<=np ; { 

world2screen(M, left^C,], left_p,G ]m , leftj^jp], &x, &y); 
left_pUD][0] = /•floorVto; 
leftjtsfjjf!] = /» floor . /(y) . 

world2scre C n(M, rightjtOJfOJ, right jtQJfl], rightjt M [2], &x, &y); 
nght_ptsfj][0] = /*floor*/(x) ; 
right_ptsUJ[l] = /•floor»/(y); 

) 
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sc = osc; 
sctPolarlnc(oinc); 
setPolarTwist(otwist); 
setPolarA2imuth(oazim); 

} 



void calcAvg (float left_pts[][2], 
float right _ptsn[2], 
float *mid_x, 
float *mid_y, 
index_st ♦ptsjndex) 

{ 

float x=0 M y=0.; 
int i; 

for (i=0; i< NUMMTHPTS ; { 
if (pts_index[i].side — LEFT^P ARAMS) { 
x += Ieft_pts[pts_index[i].pnt][0]; 
y +- left_pts[ptsjndex(i].pnt][l] ; 

} 

else { 

x += right_pts[ptsjndex[i].pnt][0]; 
y += rightj)ts[ptsjndex[i].pnt][l]; 

} 

} 

*mid_x = x / NUM_MTH_PTS; 
*mid_y y / NUM_MTH_PTS ; 

} 



void setMinMax (int *array) 
{ 
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int nf = arrayfO], i, j, k t pnt, first = l.prm; 
char temp[MAX_N AME] ; 
index_st 'indices; 
float p[2], range; 

for(i=l; i <= nf; { 
sprintf{temp/ ,0 /os.%dpnn ,, > out_basc_name, arrayfi]); 

/♦ printfC'Checking %s\n",temp);*/ 

if (IrcstorePaiarnctcnttenrp^mpJeft, temp_right,TRUE)) { 

printfT Warning: file %s doesn't exist! MVd". temp); 

if (i=first) first++; 

> 

else { 

if (i=first) {/* First file sets all max and min values! •/ 
for 0=0; j< NUM_ACTIVE_MTH_P ARAMS; j++) { 
prm = active_mth_params[j]; 

MAX_VAL(leftj>arameter[prm]) - VAL(tempJeft[prm]); 
MIN^VAL(leftj)arameter[pnn]) = VAL(tempJeft[prm]); 
MAX_VAL(right_parameter[prm]) = VAL(temp_right[prm]); 
MIN_VAL(right_parameterlprm]) = VAL(temp_right[prm]); 

} 

} 

else { 

for 0=0; j< WM_ACTIVE_MTH_P ARAMS ; j++) { 
prm = active_mth_params[j]; 

if(GT(VAL(temp_left[prm]),MAX_VAL(leftjarameter[p m ^ 

MAX_VAL(leftj)arameter[prm]) = VAL(tempJeft[prm]); 
if (LT(VAL(tempJeft[prm]), MIN_VAL{lcft_parameter[pnn]))) 

MIN_VAL(left_paramcter[prm]) = VAL(temp_Ieft[prm]); 
if (GT(VAL(temp_right[prm]), MAX,VAL(right jarameterfprm]))) 

MAX^VAL(rightj5arameter[prm]) = VAL(temp_right[pnn]); 
if (LT(VAL(tem P- right[pnn]), MIN_VAL(right_parameter[prm]))) 

MIN^VAL(right_parameter(pTm]) = VAL(temp_right[prm]); 

} 
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} 

} 

} 

printflTotal of %d kcyframesNiT, nf); 
printf("Min and Max valucs:\n \n n ); 

for 0=0; j< NUM_ACTTVE_MTH_PARAMS; j++) { 
prm = active_mth_params[j]; 

if (MAX_VAL(right_parametcr[prm]) = MIN^VAL(right_parainetcr[prm])) { 
sprintf(temp/ , %s.0.pnn w ,out_base_naxne); 
restoreParameters(temp,tempJeft, temp_right, FALSE); 
MIN^VAL(right_paramcter[prm]) = Mm_VAL(tempjight[prm]); 
MIN^VALfleft^paramcterlprm]) - MINJ/AMtempJeft[prm]); 
MAX_VAL(right__parametcr[pim]) = MAX_VAL(temp_right[pnn]); 
MAX_VAL(left_j}arameter[prm]) = MAX_VAL(temp_Ieft[prm]); 

} 

range « (MAX_VAL(Icftj>aramete^^ 
MIN_VAL(lcft_parametcr[prm]) -= range; 
MAX_VAL(leftj>ararneter(pnn]) += range; 
range^MAX_VAL(right^aram 
MIN_VAL(right_parameter[prrn]) -= range; 
MAX_VAL(right_parameter[prm]) += range; 

printfC(%d) %s: left min = %f, left max = %f\n\t\t right min = %f, right max = %An", 
prm, 

left_parameter[prm].box_label, 

MIN_VAL(leftjarameter[prm)), 

MAX_VAL(left_jjarameterQjrm]), 

MIN_VAL(right_parameter[pnn]), 

MAX_VAL(right_paramcter[prm])); 

} 

} 



int intInArray(int val, int *array) 
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for i<=n; H-f) 
if (val = arrayfi]) return (TRUE); 

return (FALSE); 

} 



float calcBeard (float p288[2]) 



float pl6[2], distanced; 
Matrix M; 

prepareMatrix(M); 

W ° r, ^ SCreen(M ' Ieft ^ &pl6[1]) . 
distance = dist (p!6,p288); 

} 

* Module: girog.h - interface for gimg.c - image mils 

* By: Ranco 

* History: 

* 13.4.94: creation. 

* 2nd version (MOVIE based): 24.4.95 

^include <stdio.h> 
^include <stdlib.h> 
^include <unistdh> 
^include <strings.b> 
#include <fcntl.h> 
#include <getopt.h> 
^include <gl/gl.h> 
^include <gl/device.h> 
^include <gl/image.h> 
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/* defines for image file manipulation */ 

#define GCMAP1 
fldefine GRGB3 
#define GRGBA4 

typedef unsigned long MPIXEL; 
typedef unsigned char PELEMENT; 

#defme R£DM OxOOOOOOff 
tfdefine GRNM OxOOOOffOO 
#defme BLUM OxOOffOOOO 
fldefine ALPM OxflOOOOOO 
#define red(x) ((x)&REDM) 
#defme green(x) (((x)&GRNM)»8) 
#define blue(x) (((x)&BLUM)»16) 
#define alpha(x) (((x)&ALPM)»24) 

typedef struct { 

unsigned short xsiz, ysiz, zsiz; 
char *buf; 

unsigned short type; 
} GimgRec, *Gimg; 

#define ImgXsize(i) ((i)->xsiz) /* num colomns V 

#dcfme ImgYsize(i) ((i)->ysiz) /• num rows ♦/ 

#defme ImgZsize{i) ((i)->zsiz) /*# color bands */ 

#define ImgRast(i) ((i)->buf) 
#defme ImgTypc(i) ((i)->type) 

define ImgPixRGBadd(i, x, y) ((i)->buf + ((yrimgXsizeOHx)) ♦sizeoiTMPIXEL)) 
#def,ne ImgPixCMAPadd(i, x, y) ((i).>buf+ ((y)'ImgXsize(iHx)) *sizeof( S hort)) 
#definc ImgPixRGB(i, x, y) ( ((MPDCEL *)(i)->buf) I^ImgXsizeOHx)]) 
#define ImgPixCMAP(i, x ( y) ( ((short *Xi)->buf) [(y)*ImgXsize(i)+(x)]) 
#define ImgInRange(i, x, y) ((x>=0.0)&&(y>=0.0)&& \ 
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(x<(doublc)IingXsi2e(i))&&(y<(double)ImgYsi2c(i))) 
Gimg GimgCreatc(shoit xsize, short ysize, int zsize); 
int GimgInit(Gimg img, short xsize, short ysize, int zsize); 
int GimgFree(Gimg img); 
int GimgDeI(Gimg img); 
Gimg GimgRead(char •!); 
int GimgWrite(Ptr f, Gimg img); 
void GimgDisp(Gimg img, short xO, short yO); 
unsigned long GimgCalcSum(Gimg img); 
void GimgClear(Gimg img); 

Gimg GimgCreateSubCopy(Gimg orig, int xO, int yO, int width, int height); 

Gimg GimgResize (Gimg image, int w, int h, int sample); 

void GimgAvg(Gimg img, long *r, long *g, long *b, int include zeros); 

void GimgMask(Gimg img, long r, long g, long b, int xO, int yOJnt width, int height); 

void GimgSetAlpha (Gimg img, long a); 

int GimgFlip(Gimg img); 

int GimgInterlace(Gimg origin, Gimg Destination); 
int GimgDelnterlacefGimg origin, Gimg Destination); 

* Module: irnage.c 

* By: Ranco 

* History: 

* 18.8.94: creation. 

^include °mv.h M 
^include "defs.h" 



extern long bgc; 

void putimage (void) 
{ 

MOVre_showFrame(movie,(MVrrame)0); 

} 



7 
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* Module: io.c 

* By: Ranco 

* History: 

* 18.8.94: creation. 

* 2nd version (MOVIE based): 24.4.95 

include "listh" 



/* Load points info from filename, into PntList PI */ 



void loadPts (char *filename, PntList *P1) 
{ 

FILE *fp; 

int i, dummy, count = 0; 
int x, y, res; 

char namc[MAX_NAME]; 
Point P; 

if ((rp - fopen (filename, V)) = NULL) { 
printfl[ M %s\n w i rilename); 
PRINT_ERR("Can't find data file\n M ); 

} 

while(!feof(fp)) { 
if ((res » fscanf(rp, "%d %d %d", &dummy, &x, &y)) != 3 || 
(dummy != count* 1)) { 
if (res — -1) break; 

PRINT_ERR("Wrong points file format!\n w ); 

} 

count++; 

} 

•PI - createPntListO; 
fseek(fp,0,0); 
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for (i » 0; i < count; i++) { 
if (fscanf[fp, "%d %d %d f \ Adurnmy, &x, &y) != 3 || 
(dummy != i+1)) { 

PRINT_ERR("Wrong points file format!\n"); 

} 

P.x = (double)x; 
P.y = (double)y; 
addPoint(*Pl, &P); 

} 

fclose(rp); 



* Module: list.h - interface for list.c 

* By: Ranco 

* History: 

* 16.2.94: creation. 



^include <stdiib.h> 
^include <stdio.h> 
#include <math.h> 
^include "defs.h" 



#defme NUM_OF^PNTS(pl) ((p])->Size) 
define POINT(pl,p) ((pl)->Pomts[p]) 
#define NUM_OF_LINES(]l) ((II)->Si2e) 
#define LINE(Il f p) ((Il)->Linestp]) 
#define LENG(l) ( ( i).i e n) 
#define SOURCE(l) ((l).pj) 
^define DEST(1) ((1).P2) 
^define XCOR(p) ((p). x) 
^define YCOR(p) (( p ). y) 



/* Some point operations */ 

define SIZE(p) (XCOR(p)»XCOR(p) + YCOR(p).YCOR(p)) 
define SetPointfr.a.b) {(p). x = (a); ^ a ^ , 
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#define GctPoint{p,r) {(r).x=(p).x; (r).y=<p).y; } 
#define AddPoints(p f pl,p2) {(p).x = (pl).x+(p2).x; (p).y = (pl).y^(p2).y; } 
tfdefine SubPoinU(p,pl,p2) {(p).x = (pl).x-(p2).x; (p).y = (pl).y-(p2).v; } 
#define MulPoint(p,c) {(p). x (fl oat ) (c); (p).y (fl oa t) ( c ) ; } 
#define DivPoint(p,c) {(p). x /= (float) (c); (p).y /= (float) (c); } 
#defme DotProduct(pl,p2) ((pl).x»(p2).x+(pl).y*(p2).y) 

Sdefine SetLine(lpl lP 2) {(l).Pl.x = (pl)->x; (l).Pl.y = (p^y; \ 
(l).P2.x = (p2>>x; (I).P2.y = (p2)->y; } 

typcdcf struct { 

float x, y; 
} Point; 

typcdef struct { 

int Size; 

Point *Points; 
} PntList_st; 

typedef PntList_st •PntList; 
typcdef struct { 

Point PI, P2; 

int len; 
} Line; 

typedef struct { 

int Size; 

Line *Lines; 

) LineList_st; 
typcdef LineList_st *LineList; 

/* Prototypes */ 

PntList createPntList (void) ; 

void freePntList (PntList 1) ; 

PntList addPoint (PntList 1, Point *p) ; 

LineList createLineList (void) ; 

void frecLineList (LineList 1) ; 

LineList addLine (LineList I, Point »P1, Point *P2); 
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/* File: mv.h 
/* 

/* Include file for the interface to movie (mv) library 
/• 

/* Written by: Ranco 
/* Creation: Wed, 5/4/95 



#ifndef_MVH 
#define _MV_H 



#include <dmedia/cl.h> 
#include <dmedia/cl_cosmo.h> 
^include <movie.h> 
^include "gimg.h" 
^include "defs.h" 
#include <audiofile.h> 

#define MOVIE ERR o 
#define MOVIE_OK 1 



typedef struct { 
char file_name[KUX_NAME]; 
MVid movie, imagetrack, audiotrack; 
int width, height; /• Size of a movie frame •/ 
int interlace; 

MVframe first, last; /* First and last frame's indices, for playback ♦/ 

MVframe num_ frames; 

MVframe cuirent_frame_no; 

double frame_rate; 

double audio_rate; 

int audio_width; 

char compression[20]; 

CLhandle compressor, decompressor, 

int cosmo_available; 

int paramBufllO]; 
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int n_params; 
Gimg cuiTcnt^frame; 

int del_img; /* a flag that indicate if current frame should be freed at the end V 
int orientation; 

int quality; /* Jpeg Quality (0 - 100) */ 
} Movie_st; 

typedef Movie_st 'Movie; 



/***•***♦• MACROS 
#define MOVIE_movieName(mv) 
tfdefine MOVIE_movieTrack(mv) 
Adeline MOVIE_irnageTrack(mv) 
fldeflne MOVIE_audiotrack(mv) 
#define MOVffi_rrameWidth(mv) 
#define MOVTE_rrameHeight(mv) 
#define MOVTE numOfFrames(mv) 
^define MOVIE_frameRate(niv) 
#define MOVIE_audioRate(mv) 
tfdefine MOVIE_audioWidth(mv) 
#define MOVIE JkstFrarne(mv) 
#define MOVIE JastFrame(mv) 



(mv->fiie_name) 
(mv->movie) 
(mv->imagetrack) 
(mv->audiotrack) 
(mv->width) 
(mv->height) 

(myGetTrackLength(mv->irnagetrack)) 
(mv->rrame_rate) 

(mv->audio_rate) 
(mv->audio_width) 
(mv->first) 



(mv->Iast) 

#define MOVIE_getCurrentFrameNo(mv) (mv->cunent_rramejio) 
tfdefine MOVIE_currentFrame(mv) (mv->currentjrame) 

#define MOVIE_loadCurrentFranie(mv) (MO VIE^etCurrentFrame(mv,mv->current_f™ 
#define MOVIE JraxneOrientation(mv) (mv->orientation) 
#define MO\rtEJraxneCompression(mv) (mv->compression) 
#defme MOVTEJramelnterlace(rnv) (mv->interlace) 
#define MOVIE JpegQuality(mv) (mv->quality) 



extern Movie movie; /* Currently allow one movie in the application! */ 



/*** Prototypes ***/ 

Movie MOYTE_open (char *filename); 
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DMparams *MOVIE_init (Movie rav, 

char *filename, 

int width, 

int height, 

double frame_rate, 

double audio_rate, 

int orientation, 

char 'compression, 

int interlace); 
Movie MOVTEjrreate (char *filename, 
int width, 
int height, 
double frame_rate, 
double audio_rate, 
int orientation, 
char *compression, 
int interlace, 
int jpeg^qualiry); 

int MOVIE_addFrame (Movie mv, MVframe index, Gimg frame, int size); 

int MOVIE_copyFrames (Movie from.mv, Movie to_mv, MVframe from_index, MVframe 

to_indcx, MVframe count); 

int MOVIE_deleteFrames (Movie mv, MVframe index, MVframe count); 

int MOVIE_getFrame (Movie mv, MVframe index, Gimg frame); 

int MOVIE_setCurrentFrameNo(Movie mv, MVframe index); 

int MOVIE_getCurrentFrame (Movie mv, Gimg frame); 

int MOVIE_addCurrentFrame (Movie mv, Gimg frame, int size); 

int MOVIE_nextFrame (Movie mv); 

int MOVIE _prevFrame (Movie mv); 

int MOVIE_showFrame (Movie mv, MVframe frame_no); 

int MOVIE bind (); 

int MOVIE_play (Movie mv); 

int MOVIEstop (Movie mv); 

int MOVIE Jiome (Movie mv); 

int MOVIE_end(Movie mv); 

int MOVIE_setStertFrame(Movie mv, MVframe frame_no); 
int MOVIE j>etEndFrame(Movie mv, MVframe frame_no); 
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int MOVIE_close (Movie mv); 

size_t MOVIE_getFrameSize (Movie mv, int frame_no); 

Gimg MOVIE_resize (Gimg image, int w, int h, int sample); 

int MOVIEjiddAudioFrame (AFfilehandle audioFile, Movie to_mv); 

AFfilehandle MOVIE_crcateAudioTrack (char *audio_filejiaine, Movie to_mv); 

#endif 

/•♦•••••♦•♦•♦•••••••♦•♦••♦♦♦•••••^ 

* Module: texture.c 

* By: Ranco 

* History: 

* 18.8.94: creation. 

#include "texturch" 
#include M x_inc.h M 
^include "general.h" 
^include "main.h" 
^include 'Vars.h" 
^include "mv.h" 

float texprops[] = { 

TXjVflNFILTER, TXJKMNT, 

TXMAGFILTER, TX_POINT, 
/* TX_MINFILTER, TXBILINEAR, 

TX_MAGFILTER, TX_BILINEAR,*/ 

TX_WRAP, TXCLAMP, TX_NULL}; 
float tevpropsf] = {TV_DECAL, TV_NULL}; 

extern int texture_map; 



void textureMapping (int op) 
{ 

long xs,ys; 
Gimg sub_img; 
int width,height; 
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if(op = TEXJDN) { 
texture_map = TRUE; 
wire = FALSE; 
if (!show_coord) { 
MOVIEJoadCurrentFrame (movie); 

} 

calcTextureCoords(left_tex,rightjex); 

width - (mt)(right_model-left_roodel); 

height = (int)(top_model.bottom_model); 

s " b _ i m g 
GimgCreateSubCopy(MOVlE_cuira^ 
height); 

zcleai(); 

if (which_image — SIDEJMG) { 
xs = ImgXsize(side_img) ; 
ys = ImgYsize(side_img); 

texdef2d(l, 4, xs, ys, (u_long •)ImgRast(side^img), 0, texprops); 

} 

else { 

GimgSetAlpha(sub_img,2551); 

texdef2d(l, 4, width,height, (ujong ♦)ImgRast(sub_tmg), 0, texprops); 

} ^ 

tevdef(l, 0, tevprops); 
tevbind(TV_ENVO, 1); 
texbind(TX_TEXTURE_0, 1); 
GimgDel(sub_img); 

} 

else { 

texture_map - FALSE; 
wire «■ TRUE; 
tevbind(TV_ENVO, 0); 
texbind(TX_TEXTUREJ),0); 
Refresh(); 
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} 

RedrawNecded = TRUE; , 

) 



void calcTextureCoords(float left_tex[][2], float right_tex[][2]) 

{ 

register int j; 

float xs, ys^.O.b^.O; 

Matrix M; 

sctModelRcgion(); /♦ Set global left/right/top/bottom_model to the 
current model's region */ 

if (which Jmage — SIDE_IMG) { 
xs = (floatXlmgXsize(side_img)); 
ys = (float)(ImgYsize(side_img)); 

} 

else { 

xs = right_modeUei\jnodel-l; /• IMPORTANT!: we assume *_model is a rectangle around 

V 

ys = top_model-bottom_model-l; /* the current projection of the model */ 
1 = left_model; 
b = bottommodel; 

) 

recompute_face(); 
prepareMatrix(M); 

for (j=l;j<= np ;j ++ ) { 

world2screen(M, leftj)t[j][0], left_ptG][l], leftj*0J[2], 

&left^texO][0], &leftjex[j][i]); 
left_tex[j][0] = (leftjex[j][0]-l)/xs; 
left_tex[j][l) =(leftJexD][l]-b)/ys; 

PRINTF5("%d: l,b-%.lf t %.lf. tex=%f, %An tt j,l f b,left.texD][0],leftjexD][l]); 
world2screen(M, right jpt[j][0], right_pt[i][l], right _j)t[j][2], 
Aright jex[i][0], ArightjexQUl]); 
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right_tex[j][0) = (right_tex[j][0]-l)/xs; 
right_tcx[j][l] = (right_tex[j][l]-b)/ys; 

} 

} 
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CLAIMS 

1 . A method for automated computerized audio visual 
dubbing of movies comprising of the steps: 

(a) selecting from the movie a frame having a 
picture, preferably frontal, of the actor's head 
and, if available, a frame with its side profile; 

(b) marking on the face several significant 
feature points and measuring their locations in the 
frame; 



(c) fitting a generic three-dimensional head 
model to the actor's two-dimensional head picture 
by adapting the data of the significant feature 
points, as measured in stage (b), to their location 
in the model; 



(d) tracking of the said fitted three- 
dimensional head model parameters throughout the 
movie, from one frame to its successor, iteratively 
in an automated computerized way and creating a 
library of reference similarity frames ; 

(e) taking a movie of a dubber wherein the 
dubber speaks the target text; 
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(f) repeating stages (a), (b), (c), and (d) 
with the dubber; 

(g) ' normalizing the dubber 's minimum and 
maximum values of. each parameter to the actor's 
minimum and maximum values for the same parameters; 

(h) mapping, on a frame to frame basis, the 
. two-dimensional actors face onto its three- 
dimensional head model by using a texture mapping 
technique, making use of reference similarity 
frames; 

(i) changing the texture mapped three- 
dimensional model obtained in stage (h) by 
replacing, on a frame to frame basis, the original 
mouth parameters with the mouth parameters as 
computed in stage (d) for the dubber and obtaining 
the parametric description for the new picture, 
with identical values to the original, except that 
the actor's mouth status resembles the mouth status 
of the dubber; 

(j) texture mapping the lips area of the same 
actor from a frame in the movie, with identical or 
very similar mouth status to the desired new mouth 
status, onto the lips area of the actor's head 
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model for the current frame and then projecting the 
lips area from the actor's head model onto the 
current new frame (an optional stage). 

2. A method according to claim i wherein the 
parameters for controlling the three-dimensional 
head model are the position, orientation, and 
expression of the head model mouth. 

3. A method according to claim i wherein the 
significant feature points on the face marked on 
stage b are the eye corners, the mouth corners, 
and the top and bottom of the face. 

4 . A method according to claim l wherein about 15 
significant feature points on the face are used in 
the tracking stage. 

5. A method according to claim l wherein the movie to 
be audio visually dubbed is a sequence of one or 
more still photographs which are identically 
duplicated, frame after frame, to create a portion 
of a movie. 

6. A method according to claim 5 for audio visual 
dubbing of still photographs in TV programs such as 
news from field correspondents. 



WO 97/15926 



89 



PCT/IB96/01056 



7. A method according to claim 5 wherein the actor 
does not speak such as a baby or a mute person. 

8. A method according to claim l wherein the original 
movie is an animated cartoon. 

9. A method according to claim 8 wherein the actor is 
an animal or any other non-human or non-living 
object. 



10. A method according to claim l wherein the movies 
are advertisment movies. 

11. A method according to claim l wherein the pictures 
of the movie and of the still photos are in 
electronic digital form. 

12. A method according to claim l wherein the pictures 
are translated into digital form, manipulated in 
digital form, and returned back to any desired 
form. 



. A method according to claims 8 wherein the 
production of an annimated cartoon is assisted by 
drawing a straight line segment for the actor's 
mouth, drawing a small actor's picture dictionary 
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containing representive reference similarity 
frames witbi completely drawn mouths, and then 
allowing these lip-line segments to be replaced 
with the coresponding lip shapes of the dubber as 
are to be found in the actor's picture dictionary. 

14. A method according to claim l wherein the "movie" 
is a painting, a drawing, or a picture. 

15. A method according to claim l for creating new 
movies or new portions of movies from a library of 
reference frames. 

16. A method according to claim l for the conversion 
of background narrative, or any spoken text, to an 
audio visual form. 

17. A method according to claim l wherein the dubber 
speaks the target text in either another language 
or the same language and the movie of the dubber is 
taken while the dubber performs a routine dubbing 
adaptation of the original into target text. 

18. A method for creation of a library of reference 
similarity frames in the method as defined in claim 
1. 
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19. A method according to claim l wherein the 
automated computerised audio visual dubbing of 
movies is done by the software, as shown in 
Appendix 1, or by similar software. 

20. A software for use in the method as defined in 
claim 1. 

21. A movie prepared by the automated audio visual 
dubbing method as defined in the preceeding claims 

22. A movie according to claim 22 wherein the movie is 
a video movie. 

23. A method substantially as herein before described 
and illustrated. 
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